I would like to exhort the questioner not to approach an acquaintance who he feels may have a problem, with one caveat. If he is a medical professional with significant training in psychiatric symptomatology and diagnosis, then he indeed has a duty to approach this person (or perhaps better still, to approach the rabbi, as discussed below.) Otherwise, I want to make you, the questioner, aware of these considerations:
*Why suppose you know whether this person is aware (s)he has OCD, and whether (s)he has already pursued treatment for it? It is possible, and indeed likely, that either the person does know, but has elected not to receive treatment--as is his human right--or, more likely, that he has already attempted (or is attempting) to seek treatment without success yet.
I would exhort you not to take it upon yourself to recommend "better" types of treatment unless you are a medical professional who is well acquainted with this person's medical history.
*Why suppose that no one else in this person's life is in a better position to observe this, and say something about it, than you are? Does this person have family, friends? Might any of them be medically literate enough to recognize his symptoms as disordered? If so, consider that people have already noticed, and are already thinking about what to do about it.
Please don't assume that "if someone had told him, he would be successfully treated"--there are a lot more obstacles to successful treatment of mental illness than you may imagine. Unless you are truly willing to take this person under your wing and provide ongoing emotional, financial, and legal support to him--which would indeed be a great reason to go ahead and speak--don't assume it is so easy and that nothing has been tried.
*Why suppose you know so well that it's OCD? In fact, the subset of brain disorders between neuropsychiatric illness and mood disorder is extremely complex. Even expert physicians have a lot of trouble teasing out the differences between OCD, OC-PD (that's "Obsessive-Compulsive Personality Disorder," a slightly different, but related syndrome), PTSD, Tourette's, tic disorder, ADHD, trichotillomania, PANDAS, autism-spectrum disorders, schizophrenia, bipolar disorder, and an array of personality and anxiety disorders too numerous to mention. It may be clear that the person is a little unbalanced, but don't assume you know it's OCD well enough that you can improve their life by telling them. Even doctors would never deliver a diagnosis only by observing symptoms.
*And now, my most important point.
Please, please consider the potential negative consequences of saying something to this person.
At best, he will simply be annoyed--and rightfully so!--that you have been watching his prayer and judging it, and that you think it is your business to point out what about his prayer is abnormal by human standards.
But most likely the effect will be much worse. Telling someone that you have noticed their behavior resembles mental illness is bound to provoke deep feelings of shame and embarrassment in anyone, mentally ill or not. To someone who indeed has a mental illness--who has probably felt alienated and/or conspicuous all his life, and who furthermore may have brain systems that are especially sensitive to rebuke--this could be damaging to the point of disaster.
I want to point out that while positive consequences of making such a remark are uncertain at best, negative consequences are a virtual certainty. The person will feel embarrassed, and most likely this embarrassment will follow them well beyond the time of the comment. I have personal experience with this disorder, and I can tell you that in my case, the type of shame that would result from such a comment would be enough to make me stop davening in that shul. I can also assure you from my experience that among those with OCD, this would not be an atypical response, and that many people without OCD would do the same.
At the very least consider whether there is another orthodox minyan within walking distance of this person before you make your remarks, but really, that is not enough. We have strenuous halachic prohibitions against embarrassing another person, even in private. (From this page: "The Sefer HaChinuch cites the Sifra on this verse that we should repeatedly rebuke someone, even several times. It then asks, “But what about if the person’s face changes from embarrassment?” No, the Midrash answers, then we do not rebuke him because the verse concludes with “and not bear a sin because of him.” In this case we are not necessarily referring to a public embarrassment, so it is clear that privately embarrassing someone is also strictly prohibited. For more, see "What about in Private?" here. For more general halachic prohibitions against embarrassing people, see the Talmud {Baba Metzia [58b-59a], Shabbos [54b-55a], Arachin [16b], and elsewhere}; the Mishneh Torah {third chapter of Hilchos Choveil u’Mazik}; the Sefer Hamitzvos {negative commandment #303}; and the Sefer HaMitzvos HaKatzar {negative commandment #79}.) This mitzvah is taken very seriously, to the point where we call embarrassment "akin to murder," and suppose that those who transgress it will lose their share in the World to Come (in Mishna [Avot 3:15] it is one of only five sins that are said to have that effect).
*So that leaves a very important question: What is to be done for this person?
First, continue to observe (only until step #4; thereafter, I would say stop observing). Second, don't gossip about it. Third, do what you can to improve this person's quality of life without having this conversation--if you see opportunities to do something for this person, take them.
Fourth, I would suggest speaking in private with the rabbi of your shul to mention the problem and ask for his help in taking care of this person. Clearly, the rabbi is in the best position to address this with the sensitivity, wisdom, and halachic correctness that are essential in such a serious matter.
If the rabbi doesn't [seem to] do anything, don't badger him about it. It may be that the best solution is in fact to let the person keep his dignity.