In all the Shuls (Synagogues) I've Davened in over the years, I noticed that all those whose parents are alive leave the shul during Yizkor. Why is this so?
Taame Haminhagim (589–590) explains (in free translation):
As Alex (thank you!) points out in a comment, the B'rachos quotation, from 20:2, is "so it should not be the case that everyone is busy [praying] and he's sitting idle" (in another context of someone who would not be praying with the community [but would, I assume, be in the synagogue]).