Shulchan Aruch Harav Orach Chaim 68:1 (basically paraphrasing the Ramo, with a reference to the Maharil):
ומכל מקום לא יעסוק בשום דבר ואפילו בדברי תורה אסור להפסיק לעסוק כל זמן שהצבור אומרים פיוטים כ"ש לדבר שום שיחה בטילה ומכל מקום מי שלומד על ידי הרהור שרואה בספר ומהרהר אין בזה איסור כלל שהרהור אינו כדבור ואינו חשוב הפסק אלא שיש לחוש להמון עם שאל יראו מן הלומד בספר ויפסיקו גם כן בדברים אחרים על כן אין לאדם לפרוש עצמו מהצבור במקום שנהגו לאמרם ויאמר אותם עמהם
Nevertheless do not occupy yourself with anything, even words of Torah are forbidden to interrupt and do the whole time that the congregation is saying Piyutim even more so any idle speech nevertheless someone who learns through thinking about what he is seeing in the book there is nothing forbidden at all because thinking is not like speaking and is not an interruption, but you should be concerned about the common people that they should not see the one learning in a book and they will also interrupt also for other things. Therefore no person should separate himself from the congregation in a place where they are accustomed to say them and he should say them with him.
Now the context there is Piyutim that some have a custom to add during Birchas Kriyas Shema, so the concern about an interruption isn't there, but the impression that it leaves on the congregation certainly is. Especially (as is explained further in that siman) where you are talking about Piyutim in Chazaras HaShatz which are not controversial (those in Birchas Krias Shema the Mechabar says to stop the custom of saying them) and common people who won't make a distinction about which words constitute the required part of the repetition and which words don't.