Perhaps an answera starting point can be provided from the Midrash in Bereishis Rabbah 44:7
אַבְרָהָם וְשֵׁם מְסֻיָּמִים בָּעוֹלָם. וְיִרָאוּ, זֶה נִתְיָירֵא מִזֶּה וְזֶה נִתְיָירֵא מִזֶּה, זֶה נִתְיָירֵא מִזֶּה לוֹמַר שֶׁמָּא תֹּאמַר שֵׁם שֶׁיֵּשׁ בְּלִבּוֹ עָלַי שֶׁהָרַגְתִּי אֶת בָּנָיו, וְזֶה נִתְיָרֵא מִזֶּה לוֹמַר שֶׁמָּא תֹּאמַר אַבְרָהָם שֶׁיֵּשׁ בְּלִבּוֹ עָלַי שֶׁהֶעֱמַדְתִּי רְשָׁעִים. קְצוֹת הָאָרֶץ, זֶה שָׁרוּי בְּקִצּוֹ שֶׁל עוֹלָם, וְזֶה שָׁרוּי בְּקִצּוֹ שֶׁל עוֹלָם. (ישעיה מא, ה): קָרְבוּ וַיֶּאֱתָיוּן, זֶה קָרַב אֵצֶל זֶה וְזֶה קָרַב אֵצֶל זֶה. (ישעיה מא, ו): אִישׁ אֶת רֵעֵהוּ יַעֲזֹרוּ, זֶה עוֹזֵר לָזֶה בִּבְרָכוֹת, וְזֶה עוֹזֵר לָזֶה בְּמַתָּנוֹת
The idea being that they were initially both fearful of the other and therefore kept their distance. Avraham was worried that Shem might resent him for killing his sons (in the war of the kings), and Shem feared that Avraham resented him for producing evil offspring / descendants. Accordingly, they dwelt in opposite parts of the world. Eventually they drew near to one and another and helped each other grow in their avodas Hashem, but until that time they kept a weary distance from each other.