So Rabbi Daniel Glatstein explores this topic here (at around 22:35 minutes in).
He brings an answer from Rav Eliezer Ginsburg Rosh Kollel, Mir Yeshiva, Brooklyn; Rav, Agudas Yisroel Zichron Shmuel and he wants to say similar to @DonielF's point that the Jewish people had three gezeiros (decrees) against them 1) that they were going to be strangers, 2) that they were enslaved, and 3) that they were afflicted. Out of these three gezeiros, they were only strangers, the other two were not relevant to them. So we take the 'Levi' and remove two-thirds as if to say these parts were not nogeiah (relevant) to them. Only the fact that they were strangers remains and that's what we keep at the seder.
However, as stated above, we are still left with the question why the Cohen matzah stays intact.