Rashi writes that one of the reasons for the plague of darkness was so that all of the wicked Jews who did not want to leave Mitzrayim could be killed and the Mitzrim would not say the Jews are also being punished like them:
ולמה הביא עליהם חשך, שהיו בישראל באותו הדור רשעים, ולא היו רוצים לצאת, ומתובשלשת ימי אפלה כדי שלא יראו מצרים במפלתם ויאמרו אף הן לוקין כמונו.
The question remains, if the reason was so that the Egyptians would not say the Jews were also being punished, then wouldn't they have noticed after the plague that there were a lot less Jews in Egypt.