This issue actually comes up in a similar situation. I'll give an example to illustrate:
Mike has considered the long path to Geirus. It's a great move, but
it takes a long time and lots of learning in the interim. His Rabbis
who are helping with this have suggested he becomes familiar with the services and have recommended his joining a minyan. There are times however, when the minyan is tight, and while he sits quietly in the back, not everyone in the minyan is aware that right now he doesn't count.
What is Mike to do? It's embarrassing every time to tell the person
that he doesn't count. It's not nice to be considered out of the
fold, when for all intents and purposes, he looks Jewish!
That is why Mike makes a point of only Davening in the main Minyanim
in the Shul - a sure bet that there'll be 11. The Gabbai, who gets
all sorts of requests and requirements knows the situation, and
handles the issue subtly and with respect and sensitivity and avoids embarrassment. He tell's the minyan that we don't have the required 10. The person in the back already davened, and the Rav's psak is that since there is a later minyan anyway, they should rather wait till then to daven.
Mike has been told by his teachers that he should inform the Gabbai,
and if, he is invited for Shabbos, that he may not touch non-mevushal
wine.
You write:
It's probably best to explain the situation to the gabbai, but I can't think of a good way to do so. There's a very high risk of causing embarrassment for everyone.
I think the issue is more to do with how to handle a delicate situation sensitively. This can easily be done without embarrassment. People in leadership roles are supposed to be helpful and sensitive and there is no reason to think otherwise here. If not, perhaps this is the congregation that the person should avoid.
Obviously one should not feel embarrassed by the situation - these things are often no-one's fault but it is what it is. Causing harm, especially when discussed in private is hardly a need for concern. Rabbis have thick skin. There are many cases that are often far worse than what actually happens practically.
You write:
If he's a convert, it'll necessarily force others to remind him of his past, which is forbidden.
You're right it is forbidden to remind him of his past, because you're effectively shaming him. He is now a new person, and you're shaming him.
However, in this context: privately, with a desire to do the right thing is hardly a cause for being shamed. A convert should be proud that he has chosen to enter the Covenant of Avrohom Avinu by his own choice! In the context of doing the right thing, and privately discussing an issue should never be a cause for humiliation, shame and embarrassment.
I would speak to the Rabbi of the Congregation and explain the situation in detail. Often there are things that people miss or are not aware of. If the person concerned is approached by the Gabbai, or a congregant, the best is probably to deflect and say: "I've already davened" or "I can't daven right now for personal reasons". If this would cause embarrassment, then it seems from a similar answer presented in mi yodeya, that there would only be an issur d'rebannon, and that the shaliach tzibur should be told that he should have in mind that the tefillah is a tefillas nedava. I would however speak to the Rabbi first, or speak to a Local Orthodox Rabbi. Geirus, Jewishness and the issue is complicated, sensitive and controversial. Speaking to someone about it in a private setting with enough time can be worked out. I think this is the key - making judgments and assumptions before actually speaking to someone in a leadership role is foolhardy. No-one should ever feel embarrassed about preventing harm or doing the right thing!
You write:
Telling others or letting them assume he counts also has a problem of lifnei iver, since it can result in an invalid (according to B) minyan or aliyah. Telling others he doesn't count is an outright lie, and an 11th man might not come along.
There is a machlokes as to whether or not Lifnei Iver applies to all Issurei D'Rebnonan. In this context, Davening in a minyan is at best a minhag, possibly D'Rebonan. In fact, since we don't know who is Jewish and who isn't today - we only base ourselves on Chazaka - the principle that what was is assumed to be the case, nonetheless, Hashem rewards and makes the effort to help His children do the right thing.
Perhaps it's best to contact the rabbi beforehand? But it still causes embarrassment, though at least it's private. And it doesn't cover the case where the gabbai finds him and asks him to complete a minyan.
You answered your own question - Yes should contact the Rabbi beforehand. It's only embarrassing if it is made to be - and I hardly expect a person in a leadership role - given time and adequate information and warning shouldn't be able to help. You can certainly ask the Rabbi what to do in the case the Gabbi finds him. Rabbis are good at fielding questions!