Of particular interest are the prayers and the order of reading the psalms in Nusach Ashkenaz.
So in the Nitei Gavriel on Aveilus (cheilek alef), chapt 10 it lists the halachos of those doing shemira. The following points are pertinent to your question:
(ois alef) One who is doing Shemirah is exempt from krias Shemah and any tefillah. (Refer to SA OC 71:3, SA YD 341:6 and SA YD 403:9). Therefore one can arrange, if possible, that there be two shomrim, so that one can guard whilst the other goes and says Shemah and davens. This idea I also saw brought down in the first cheilek of Gesher HaChaim, perek hei.
(ois ches) Even though it is forbidden to to daven and say divrei Torah in front of the deceased, it is permitted to say tehillim. (This idea is also brought in Gesher HaChaim linked above). Even an onein can say tehillim (this is further elaborated later here (ois gimmel)) as they are doing it to respect the deceased. However others are stringent since an onein by definition is patur (exempt) from doing any mitzvos.
So to answer your questions. General tefillos are avoided unless there is someone else there with you so that you can go and daven elsewhere whilst they guard. And as far as tehillim, one can recite (and I have read elsewhere that it is encouraged) but there is no set standard or formal selection.
However there is sefer which is cited by both the Nitei Gavriel and Gesher HaChaim entitled Ma'avar Yabok by Rabbi Aharon Berachia of Modina which provides select sections from throughout tanach (incl. Tehillim) that one can work through.