In the Yom Kippur service during Temple times, the Kohen would (Lev 16:7-10ff):
And he shall take the two goats, and set them before the LORD at the door of the tent of meeting. And Aaron shall cast lots upon the two goats: one lot for the LORD, and the other lot for Azazel. And Aaron shall present the goat upon which the lot fell for the LORD, and offer him for a sin-offering. But the goat, on which the lot fell for Azazel, shall be set alive before the LORD, to make atonement over him, to send him away for Azazel into the wilderness. (mechon-mamre.org)
הִקְרִיב אַהֲרֹן אֶת-פַּר הַחַטָּאת, אֲשֶׁר-לוֹ; וְכִפֶּר בַּעֲדוֹ, וּבְעַד בֵּיתוֹ. ז וְלָקַח, אֶת-שְׁנֵי הַשְּׂעִירִם; וְהֶעֱמִיד אֹתָם לִפְנֵי יְהוָה, פֶּתַח אֹהֶל מוֹעֵד. ח וְנָתַן אַהֲרֹן עַל-שְׁנֵי הַשְּׂעִירִם, גֹּרָלוֹת--גּוֹרָל אֶחָד לַיהוָה, וְגוֹרָל אֶחָד לַעֲזָאזֵל. ט וְהִקְרִיב אַהֲרֹן אֶת-הַשָּׂעִיר, אֲשֶׁר עָלָה עָלָיו הַגּוֹרָל לַיהוָה; וְעָשָׂהוּ, חַטָּאת. י וְהַשָּׂעִיר, אֲשֶׁר עָלָה עָלָיו הַגּוֹרָל לַעֲזָאזֵל, יָעֳמַד-חַי לִפְנֵי יְהוָה, לְכַפֵּר עָלָיו--לְשַׁלַּח אֹתוֹ לַעֲזָאזֵל, הַמִּדְבָּרָה.
These two se'irim are treated much the same way that Abraham treated his sons - one (Isaac) was sacrificed, and one (Ishmael) was sent into the midbar.
I know that Rashi (and maybe Talmud?) says that se'ir l'Azazel is thrown off a cliff, but the plain meaning is that it's sent to the midbar. I'm less concerned with the technicalities than the similarities between these two. (Also note that these are the leynings from both days of Rosh Hashannah and Yom Kippur.)
Why do we mimic Abraham's fathering? Alternatively, why did Abraham perform the Yom Kippur service? What do we learn from this? Are there any sources that make this parallel (aside from my own thoughts and later a painting I saw at the American Visionary Arts Museum in Baltimore)?