My office is situated right next to a Jewish cemetery. A coworker, noticing the yarmulke on my head, asked me why he sometimes sees stones placed on some of the gravestones. I told him that as far as I know the reason is to show respect for the deceased by showing that someone was there visiting the grave, but I'm wondering if there's a better or deeper reason. I've asked a few locals including the rabbi of my shul, but I have yet to find a good answer.
A less poetic but more probable explanation than the one SimchasTorah linked to...
Dates back to when grave markers were cairns, which is the biblical meaning of the word "matzeivah" (before we shifted it to mean tombstone). A cairn is a pile of stone. With rain and wind, the pile would shrink. So, out of respect for the deceased, so they not be forgotten, visitors would enlarge the pile of stones by adding to it.
From the Be'er Heiteiv, Orach Chaim 224:8:
מה שתולשין עשב או צרור ומשימין על מצבה אינו אלא משום כבוד המת להראות שהוא היה על קברו
That people pick grass or a stone and place it on the grave marker is simply [to accord] honor to the deceased; to show that one has been present at the grave.
Quoted in the "laws" section of the OU/Artscroll Siddur for the House of Mourning.
I heard this morning from Rabbi Shmuel Tendler Shlita - Rabbi of Sons of Israel in Lakewood, that the word Even is an acronym for Av, Ben, Neked or alternatively Eim, Bas, Nekda. That is why we place a stone as we are saying we are a continuation of you. (This only explains why one would place on the grave of a parent or grandparent)
While not written from a Halachic perspective, the website of a Jewish cemetery organization in Woburn, Ma, USA, includes a story of a personal visit to family graves which includes the explanation of the old custom of building cairns as the roots of the modern tradition of leaving a stone; the cemetery even provides a basket of stones for those who have not brought their own.