Is it necessary for a Kohen to clarify that a prospective match who is a ba'alat teshuvah has not had sexual relations with a non-Jew before dating her, or can one rely on a chezkat kashrut? If the former what steps need to be taken to clarify her status?
My understanding is that many people today will start off with the assumption that a woman who bacame baal teshuva after a certain age is just not kohen-eligible.
As for your question -- it's not an easy matter, but if it's prohibited it's prohibited. ("Don't embarrass someone" doesn't mean I can ignore the serious possibility of halachic prohibitions.)
Most shadchanim simply have a checkbox -- "are you ineligible to marry a Kohen?" The lady can simply check this -- or say "sorry no Kohanim" without giving another drop of detail. Could be her father wasn't Jewish, could be she had a non-Jewish boyfriend, could be rape, could be all sorts of things -- none of it is the shadchan's business.
So while the kohen asking her point-blank about her past in graphic detail is uncouth, there are better ways of doing it -- for instance, involving a rabbi or a shadchan, and handing her a form that says "if any of the following apply, check this box." That's obtaining the minimally-necessary knowledge in the least-hurtful way.
Have a look in the Tosafos בתולה שעיברה in חגיגה on :דף י"ד
Essentially Tosafos there brings a possible rule that if the wife would also be transgressing (in that case והוא אשה בבתוליה יקח) then we can rely on her. As opposed to Yichus issues where she really has nothing to lose if she lies.
So we may to able to answer your question thus:
As long as the potential wife is aware that they would both be living in sin if she had sexual relations with a non-Jew before dating him, we could supposedly rely on her unwillingness to do so (especially since she's a Ba'alat Teshuva, who has turned around her life not to live in sin) and there would be no need to ask her.
If this kohen is himself a BT can we not say he is also most likely a 'cholol' and therefore permitted to marry any other BT. I would say that if he is already married to a BT and there was a good chance that his mother was also a zonah he would be able to stay with her. We see often in the gemoro that one tries very hard not to separate a man from his wife. ניסת לא תצא. It is not korus only a 'lav' where we are even more 'meikal'. Also today kohanim dont have a real chazaka.