The short answer to this question is that the midrashim read history backwards.
That is, since we know, for example, that Yishmael is not chosen over Yitzchak to be the "carrier" of God's blessing and promise to Avraham, the author of the midrash assumes that there must have been something undeserving in him or he must have done something wrong. Therefore, they will try to "read into the text" wrongdoings or flaws of Yishmael to justify his destiny.
The same is true with Esav. We know that Esav was not chosen (or even "co-chosen", like the tribes) as Yitzchak's primary progeny; in fact, God even says, " הֲלוֹא אָח עֵשָׂו לְיַעֲקֹב נְאֻם-יְהוָה וָאֹהַב אֶת-יַעֲקֹב וְאֶת-עֵשָׂו שָׂנֵאתִי" (Malachi 1:2-3) - God hated Esav. "But why?", wonders the midrash. He must have been a bad person to deserve God's hatred. So the midrash will not hesitate to interpret slight nuances in the text or come up with back-stories that explain Esav's evildoing.
Of course, this holds in reverse as well. Yehuda is known to be one of the twelve tribes to inherit the promise of Avraham and all the blessings of God that came with it. In fact, history gives Yehuda's descendants a large slice of the pie of prosperity. Again, why? With the belief that God is just, the midrash needs to explain this. So it will judge Yehuda extremely favorably under the assumption that he must have been righteous. Anything that looks bad in the text is reinterpreted or downplayed, and anything good is emphasized and glorified.
This sometimes leaves the uninitiated reader of the midrash perplexed or even put-off. But with this understanding, the midrash tends to make much more sense.