When a person says "HaKel HaKadosh" instead of "HaMelech HaKadosh" during Aseres Yemei Teshuva the halacha is they must go back to the beginning of the tefillah and start over again. What would be the din of the person didn't go back to the beginning but rather said the bracha of "Atah Kadosh" over again with "HaMelech HaKadosh" and then continued davening normally. Would this be ok b'dieved or would they have to start the whole tefillah over again after finishing?
This is not a special עשרת ימי תשובה law, but a general law that the first three Brachos are seen as one unit. The last three as well.
If one makes a mistake in a Bracha he needs to return to the beginning of that Bracha. If a mistake is made in one of the first three - he returns to the beginning of the three, which is the beginning of 18.
(EDIT) Hence, if one fails to fix the Bracha which was messed up - the whole Tfila should be said again.
Rambam, Tefila 10:1:
מי שטעה באחת משלוש ברכות הראשונות חוזר לראש ואם טעה באחת משלוש אחרונות חוזר לעבודה ואם טעה באחת מן האמצעיות חוזר לתחילת הברכה שטעה בה
A friend pointed out Rav Ovadia saying that a shatz during chazara should return to third bracha. However, he explains this is because during chazara the 3-bracha-unit ends after kedusha, so there's no need to go to beginning. A yachid should go to beginning. Other poskim do not agree with this psak (I've seen Rav M. Eliyahu explicitly).
(P.S. The friend argued that if it was possible to return to 3rd bracha only that's what the psak would be since we make a bracha levatala (on the first two brachot) doing otherwise. I'm unsure about this point.)
The KafHaChiam (יעקב חיים סופר) says (582:9) that as long as you have not started the next Bracha, בדיעבד, you could still change and say המלך הקדוש - and there’d be no need to restart the Bracha, just fix it.
ט) נראה דבדיעבד אם שהה יותר מכדי דיבור יכול לומר המלך הקדוש כל זמן שעדיין לא התחיל בברכה שלאחריה משום ספק ברכות אבל אם לא נזכר עד שהתחיל בברכה שלאחריה אפי' תיבה אחת צריך לחזור לראש וכמ"ש באו' שאח"ז.