The source for Rashi is the Sifra - Toras Kohanim which states:
מנין שלא היתה אומה באומות שהתעיבו מעשיהם יותר מן המצריים ת״ל כמעשה ארץ מצרים לא תעשו. מנין לדור אחרון שהתעיבו מעשיהם יותר מכולם ת״ל כמעשה ארץ מצרים לא תעשו. מנין למקום שישבו בו ישראל שהתעיבו מעשיהם יותר מכולם ת״ל אשר ישבתם בה לא תעשו . ומנין שישיבתם של ישראל גרמה להם לכל המעשים הללו ת״ל אשר ישבתם בה לא
From where do we see that there was no nation as morally corrupt as the Egyptians? The verse (Leviticus 18:3) teaches, "According to the ways of the land of Egypt... you shall not do." From where do we see that the last generation was more corrupt than all of them? "According to the ways of the land of Egypt... you shall not do." From where do we know that the place where the Jews settled was more corrupt than all of them? ""That you dwelled in, you shall not do." And from where do we know that the settling of the Jews caused them to engage in all these acts? "That you dwelled in do not do."
This would appear to indicate that Goshen only became the most immoral location in Egypt after the Jews settled there, and that it became progressively worse while they were there, so that they sunk to their lowest depths during the last generation before the Exodus. The Sifra concludes (although the Malbim understands the conclusion differently) that it was the presence of the Jews that caused this moral descent. The Rash M'Shantz and Raavad both write that the reason for this is that the sin of enslaving the Jews caused them to sink progressively lower - for עבירה גוררת עבירה - one sin brings another. (It may be worth mentioning that slavery has often been noted for having a deeply unhealthy effect on the morality of the slave-owners.)
The הגהות מהרי"ד adds (along the lines of Yehuda's answer) that there is a natural reaction of impurity against holiness, so that when the Jewish people dwelt in Goshen, the higher level of holiness caused a parallel increase in impurity in the surrounding society. (Personally, I am not terribly satisfied with this approach for several reasons. On a peshat level, my main criticism is that, according to this approach, as the Jewish people fell under the influence of the surrounding society (as many sources indicate), this should have caused the rise in immorality to abate, yet the midrash indicates that the last generation was the worst of all!)