In all the Shuls (Synagogues) I've Davened in over the years, I noticed that all those whose parents are alive leave the shul during Yizkor. Why is this so?
Taame Haminhagim (589–590) explains (in free translation):
If he remains there, he might say it with them, and there's a covenant made with the lips [that what they say is true comes true]. Another reason is that everyone's busy saying Yizkor and he's silent, and it says in B'rachos that it should not be the case that all are busy etc. There's a slight concern of ayin hara in such a case (as in Y'vamos 106, "Do you have parents in town?" ―"Yes", and he set his eye upon them and they died).
As Alex (thank you!) points out in a comment, the B'rachos quotation, from 20:2, is "so it should not be the case that everyone is busy [praying] and he's sitting idle" (in another context of someone who would not be praying with the community [but would, I assume, be in the synagogue]).