What was the reasoning of the 'scorners of the generation'?

Question

Bereshit 25:19 reads:

וְאֵלֶּה תּוֹלְדֹת יִצְחָק בֶּן אַבְרָהָם אַבְרָהָם הוֹלִיד אֶת יִצְחָק:

And these are the generations of Isaac the son of Abraham; Abraham begot Isaac.

Rashi: (bold in the English is mine)

אברהם הוליד את יצחק: ... לפי שהיו ליצני הדור אומרים מאבימלך נתעברה שרה, שהרי כמה שנים שהתה עם אברהם ולא נתעברה הימנו, מה עשה הקב"ה, צר קלסתר פניו של יצחק דומה לאברהם, והעידו הכל אברהם הוליד את יצחק, וזהו שכתב כאן יצחק בן אברהם היה, שהרי עדות יש שאברהם הוליד את יצחק:

Abraham begot Isaac: ... Since Scripture wrote: “Isaac the son of Abraham,” it had to say: “Abraham begot Isaac,” because the scorners of the generation were saying that Sarah had conceived from Abimelech, for she had lived with Abraham for many years and had not conceived from him. What did the Holy One, blessed be He, do? He shaped the features of Isaac’s face to resemble Abraham’s, and everyone attested that Abraham had begotten Isaac. This is the meaning of what is written here: “Isaac, the son of Abraham,” because here is proof that “Abraham begot Isaac.” - [From Midrash Tanchuma, Toledoth 1]

Why would the scorners think that? I suppose it was well known that Avraham already had a son (Yishmael) from Hagar - so wouldn't that fact make it clear that it was Sarah that was barren?

So then I thought: 'Well, that's why they are scorners - they don't need to be logical'

But then, why would Hashem pay attention to illogical nonsense?

Answer 1

The K'li Yakar suggests that later scorners would argue additionally that one of Yitzchak's sons was evil, so the great Avraham could not be Yitzchak's father. (This is why, says the K'li Yakar, the next verse points out that Rivka's brother was the wicked Lavan, and it was from her family that Esav inherited his nature). To preempt this, Yitzchak's appearance was made to resemble Avraham.

As far as your final question, I heard somewhere (if someone could find a source it would be much appreciated) that the nature of scoffery is that it is contagious, yet impervious to logic. Therefore, the only way to undermine the nonsensical slander was with a clear sign that would be visible to everyone - the resemblance between Avraham and Yitzchak (see also Bava M'tzia' 87a, היו כל אומות העולם מרננים ואומרים ראיתם זקן וזקנה שהביאו אסופי מן השוק... ועדיין היו מרננים ואומרים אם שרה הבת תשעים שנה תלד אברהם בן מאה שנה יוליד מיד נהפך קלסתר פנים של יצחק ונדמה לאברהם פתחו כולם ואמרו אברהם הוליד את יצחק ; Bava Kamma 97b, ואיזהו מטבע של אברהם אבינו זקן וזקינה מצד אחד ובחור ובתולה מצד אחר).