What should one do if he forgets all the changes in the Tefilla during the Aseret Yemei Teshuva? When should he go back, when may he continue?
If one forgot to say anything but HaMelech HaKadosh at the end of the third blessing, he need not go back for it. (Shulchan Aruch Orach Chaim 582:5). If he has not yet said "Baruch atah Hashem" at the conclusion of the blessing, he can go back to the addition.
If one forgot to say "HaMelech HaKadosh", and instead concluded "HaEl HaKadosh" as usual, if he does not immediately correct himself he must begin anew. Because the first three blessings are one unit, he must begin from the beginning, and can't merely repeat the one blessing said incorrectly. (ibid 582:1)
If instead of saying "HaMelech HaMishpat" he said the usual ending for that blessing, he should go back to that blessing if he has not yet taken his steps back at the end of the amidah. If he already took his steps back, he must repeat the Amidah (ibid). Rema writes that if in his ending he included the word "Melech" he has fulfilled his obligation post facto. (ibid. Glosses to 118:1)