Let's assume the question is the color of the garment, not the stripes. There is actually some (small) halachic basis for such an argument. (In addition to whatever "soft" concerns about distraction, disruption, or the like.)
Rambam's opinion is that if the whole tallit is pink/red/grey/yellow, then the strings (except fo the techelet one) should be pink/red/grey/yellow as well. Other rishonim say no, the other strings are always white. The ashkenazic practice is that they're always white, however in an attempt to accommodate the Rambam's opinion, many prefer to stick with a white-garment tallit, so that everyone would agree the strings should be white.
There's a recording where Rabbi Hershel Schachter says "I saw someone wearing a grey-garment tallit, we should tell him that the custom of our yeshiva is to only wear white ones, because of this opinion of the Rambam!"