It depends on where you are.
If you are saying Shema, the Shulchan Aruch 64:2-4, citing the Gemara in Berachos 16a, discusses what to do in three different instances:
- If you know what paragraph of the Shema you are in, but you're not sure what words you have read, you should go to the beginning of the paragraph;
- If you are between parapgraphs, but can't remember if you're between the first and second paragraphs, or the second and third, you should resume from the top of the second paragraph;
- If you are at the word "uksavtam," but you're not sure you're in the first paragraph or the second, you should continue from the one in the first paragraph.
The Mishna Brurah (64:7-11) adds:
- If you are unsure where in the first paragraph you are, but you're sure you said the Shema and the Baruch shem kavod, then you should begin at V'havta;
- If you know you said a particular word in the first paragraph, but you can't remember what you read thereafter, you should go back to that word;
- If you can't remember which paragraph you finished, but you know you read a particular word in the second paragraph, then you should return to "Vayomer."
- Based on the comment of the Taz, the M.B. adds that if you know you said "uksavtam" but can't remember which paragraph you read it in, you would go back to the word in the first paragraph; and
- If you were davening Shema in a shul, and the shaliach tzibur is in the second paragraph, we say that is sufficient evidence that you made it to the second paragraph and can continue from there, rather than going back to the first paragraph.
The Gemara in Berachos 21a, discusses what to do vis a vis the Shemoneh Esrai. The Gemara says that it is a machlokes whether Shemoneh Esrai is a pure suffeik d'rabbanim and therefore you would not say it again, or if there is a special inyan that you can daven voluntarily, even when you're not in doubt, and simply repeat the prayer as a voluntary prayer. The Shulchan Aruch, 107:1, holds that the halacha follow that one should repeat, reasoning that if one did say the Shemoneh Esrai before, his new prayer is a voluntary offering, and if had not said it, then the new prayer is a obligatory offering. The M.B. (107:2) opines that one should not, in theory, repeat the Shemoneh Esrai unless he can think of different applications for the 13 requests in the middle section of the prayer, when there is doubt about having said it, then we regard the repetition as a "new" prayer, even if one actually did say it before.
My rav, Rabbi Dovid Rosenbaum, shlita, recently spoke on the issue of whether one is unsure whether he skipped over some of the brachas or made a mistake in the Shemoneh Esrai. Applying the principles stated above regarding the Shema, he said that if you have not gotten to "v'al ha kol," one should go back to what you remembered having prayed and restart from there. The S.A. (117:2-5) provides rules regarding fixing mistakes in the saying of the "v'sain tal u'matar" which generally require the repetition of the entire prayer if you have already backed out, or allow you to restart from birchas hashanim depending on the season and other factors. A detailed discussion of these halachas can be found here.