I want to point two details in your question.
- The fact that you know that one who wears RTT can be perceived as arrogant,
- and the fact that you are in a community (lituanian) in which a large majority of men wear Rashi's Tefilin only. So your question is not for, e.g. a young Chasidish Avrech who started to wear RTT after marriage. He is perfectly conformist and without doubt not arrogant. He makes nothing new, nothing that we can not expect. Perhaps the contrary would be strange.
Shulchan Aruch OC 34, 3:
לא יעשה כן אלא מי שמוחזק ומפורסם בחסידות:
Needs to do this only a person recognized for his devotion.
From the Bet Yosef and Darke Moshe in the Siman we learn tha this statement is extracted from a Shut Maharil. The issue is to wear the two, RTT and Rashi T. together. This cannot may be discreet. So the word of Bet Yosef regard the problem of a public wearing of Rabenu Tam Tefilin. In this case we can understand why the Baer Heytev adresses again the question. The BH addresses an other practice: to wear the RTT at the end of the prayer, alone, after Removing Rashi Tefilin.
נשאלתי על איש אחד שהיה נוהג להניח תפילין של ר"ת לאחר התפלה בפרהסיא בפני הקהל אי מיחזי כיוהרא. פסק בתשובת מהר"ש הלוי דמיחזי כיוהרא וצריך שיבטל מנהגו. וכ"כ בתשובת שבות יעקב ח"ב סי' מ"ד שאפי' אם מקצת עושין יש בו משום יוהרא ואם מניחן בפני אדם גדול שאין נוהג להניחם כי אם בקרב ביתו ודאי מחזי כיוהרא ע"ש.
I received a question about a man, who used to wear RTT after the prayer, in front of everyone. is it considered as a manner to brag?
The answer in Teshuvat Maharash Halevi is that this is regarded as a kind of boasting, and he hav to stom this habit. Ther is a similar answer in Teshuvot Shevut Yaakov, part II, number 44, and he added that even if he is not alone, but few people make this, there is a problem of boasting. Additionally, if he wear RTT in front of a great man who prefer to wear them at home, there is obviously a problem of boasting.
We learn that the problem of brag is only in front of other people who has not this habit. The fact to wear RTT at home doesn't present problem of Yuhara, but, the chumra needs to be proportional to the general level of the person.
It seems from the above shut reporting, that the context is very important. For example, nowadays in Israel in a shtible, if you see someone wearing RTT, everyone will think that he is chasidish or in any group who has the minhag to wear RTT, he only makes as his friends and / or family. You can judge the thing only in the context you are. E.G. if a given morning, in a lituanian Yeshiva, a lituanian Bochur decide to wear RTT in front of the Rosh Yeshiva, the question of Yuhara will be actual.
purchasing and wearing tefilin shel Rabenu Tam outweigh the risk of appearing arrogant ---> no, if it is made discretly.