Rav Moshe Shapira explained this as part of a broader approach to the difference between the philosophy of Avraham and that of Shem.
To summarize, Shem was an ascetic, and believed in spirituality divorced from physicality. His Torah was the Torah of pure spirituality which does not involve the physicality. He is, therefore, the authority on the disconnect between spiritual heights and physical depravity. Therefore, Yaakov visited the Yeshiva of Shem and Ever on his way to Lavan (Rashi to Bereishis 28:11) - a Yeshiva in which one does not sleep (neglect of the body) - in order to understand the Torah of disconnected spirituality, as Lavan had a lofty neshama and was a miserable wretch.
Avraham, on the other hand, believed in spirituality which manifests itself in your every day life. His name was 248 in gematria corresponding to the 248 limbs which are involved in serving Hashem. Nimrod had Avraham thrown into a furnace but left Shem alone because Avraham's was a threatening philosophy but Shem's spirituality did not infringe on how they wanted to live their lives.
Rivka had a pregnancy which she believed to be one individual with very polar opposite pulls. When she walked by a Beis Medrash, Yaakov would start squirming to come out, and when she passed by a house of idolatry, Eisav would squirm to come out (Rashi to Bereishis 25:22). But she did not know that this was two diametrically opposite babies inside her until after she was given that answer - she thought it was one child with an extreme disconnect between his spirituality and the impact it would have on how he lived his life. Therefore, Shem was the Navi to consult.