The following is a quotation from an essay I wrote addressing this issue:
In a famous medrash, the Sages tell us that in addition to functioning as a punishment for the Egyptians, the plague of darkness also served to hide an event that God did not wish the Egyptians to witness. The medrash states (Shemos Raba 14:3, also see Tanchuma, Va'era 14, and Tana D'Vei Eliyahu 7):
Why did He bring darkness upon them? Blessed be the Name of the Holy One, blessed is He, for there is no favoritism before Him, Who delves deep into the [human] mind and examines [their] thoughts: For there were sinners (פושעים, Rashi here has רשעים, "wicked people") in Israel who had patrons among the Egyptians, and they had wealth and honor there, and they did not wish to leave. God said, "If I bring a plague upon [these sinful Jews] openly and they die, the Egyptians will say, 'The same thing that is happening to us is also happening to them!'" Therefore He brought the darkness upon the Egyptians for three days, so that [the Jews] would be able to bury their dead, and their enemies would not see.
There were Jews who were comfortable in Egypt and did not want to leave. The Sages do not accuse these people of any kind of sinful behavior, whether towards God or to their fellow men. Their entire wickedness is summed up in the fact that they did not wish to leave Egypt!
Moreover, we know that there were some genuinely wicked Jews who did leave Egypt together with their fellow Jews, the best known examples being the famous troublemakers, Dathan and Abiram. And the Jewish people as a whole were far from perfect. The Sages tell us that when God drowned the Egyptians at the Red Sea, the angels challenged the Divine Justice in drowning the Egyptians and not drowning the children of Israel, famously declaring, "הללו עובדי עבודה זרה והללו עובדי עבודה זרה" - "These are idol worshipers and those are idol worshipers!"
The implication of this medrash is that the issue was not that these particular Jews were exceptionally sinful, but rather that they were sinful in a very specific way, one which made it impossible for them to live to see the redemption of their people from Egypt. This was the simple fact that, regardless of any other virtues they may have possessed, these Jews wanted to stay in Egypt. They were quite comfortable in Egypt, and they saw no reason to leave!
As Jews, we have an obligation to see ourselves as part of a larger unit, the Jewish people, to which we are intrinsically connected. Our connection to God as individuals cannot be distinguished from our connection to the Jewish people as a whole. We can never simply go our own way, as if the fate of our fellow Jews means no more than that of any other random group of people.
From my blog post, Do Not Separate from the Community.
As an aside, I would add that Rav Shimon Schwab argues (convincingly, in my opinion) that the medrash regarding the large numbers of Jews who died during the plague of darkness is not intended to be taken literally.