My office is situated right next to a Jewish cemetery. A coworker, noticing the yarmulke on my head, asked me why he sometimes sees stones placed on some of the gravestones. I told him that as far as I know the reason is to show respect for the deceased by showing that someone was there visiting the grave, but I'm wondering if there's a better or deeper reason. I've asked a few locals including the rabbi of my shul, but I have yet to find a good answer.
A less poetic but more probable explanation than the one Isaac Moses linked to...
Dates back to when grave markers were cairns, which is the biblical meaning of the word "matzeivah" (before we shifted it to mean tombstone). A cairn is a pile of stone. With rain and wind, the pile would shrink. So, out of respect for the deceased, so they not be forgotten, visitors would enlarge the pile of stones by adding to it.
From the Be'er Heiteiv, Orach Chaim 224:8:
Quoted in the "laws" section of the OU/Artscroll Siddur for the House of Mourning.