I've seen in halacha concerns for people making extremely far-fetched negative conclusions. The following are but two examples:
In Shulchan Aruch Orach Chaim 3:13 it says:
לא ישתין מעומד מפני ניצוצות הניתזין על רגליו
One should not urinate from a standing position lest it sprinkle down upon his legs
The Mishnah Berurah explains from the gemarra in Niddah 13a that if it sprinkles on his legs, people will think he has an injury, making him infertile. They will then conclude that his children aren't his, but rather his wife committed adultery, making the children mamzerim.
A second example is Shulchan Aruch Orach Chaim 128:5, which says:
לא יעלו הכהנים לדוכן במנעלים אבל בבתי שוקים שרי:
Kohanim may not ascend to the platform in shoes, but in socks it is permitted.
The Mishnah Berurah explains from the gemarra in Sotah 40a that if we allow them to wear shoes, maybe one of their shoes' laces will come undone, and it will be embarrassing for them to go up. They'll subsequently spend time tying them, missing the opportunity to bless the people. People will notice that he didn't go up, and suspect him of not being a kosher Kohen.
These are but two examples of concerns that people will make extremely far-fetched negative conclusions. On the other hand, I've encountered the opposite. The gemarra assumes people will make extremely far-fetched positive conclusions.
For example, Yevamos 15a is discussing the prohibition of lo sisgodedu, where you can't have splintered groups of Jews together following different rulings. The gemarra brings an example from Shammai, who followed his own opinion, when it would seemingly be an issue of lo sisgodedu.
מתיב מר זוטרא מעשה וילדה כלתו של שמאי הזקן ופיחת את המעזיבה וסיכך על גבי מטה בשביל קטן ש"מ עשו התם הרואה אומר לאפושי אויר קעביד
§ Mar Zutra raised an objection to this issue from a different source: There was an incident in which the daughter-in-law of Shammai the Elder gave birth to a son. In Shammai’s opinion this newborn baby is immediately obligated in the mitzva to sit in a sukka, and he therefore removed the mortar [ma’aziva] covering the ceiling and he placed sukka covering over the bed for the minor. Conclude from here that Beit Shammai did act in accordance with their opinions. The Gemara answers: There is no proof from there, as anyone watching would say that he did it merely to increase the air. Since people would not necessarily think that he removed the mortar as a ruling of halakha, this behavior is not considered the formation of a faction.
It is very far-fetched to assume that in the middle of Sukkos, Shammai removed the mortar and put up schach over a child, when we know his opinion is children that young need to be in the Sukkah, that he did so only to make the room "more breezy". We see that we go very far to avoid suspecting people of wrongdoing (lo sisgodedu), yet in the earlier examples people aren't assumed to judge so favorably.
Does anyone address this seeming contradiction? It doesn't have to be these specific examples. Just a source which addresses far-fetched positive and negative conclusions.