In Bereshit 22:3, Rashi tells us that when Avraham went to sacrifice Yitzchok, the two lads he took with him were Yishmael and Eliezer.

Wasn't Yishmael already gone from Avraham's house? In Bereshit 21:21 we are told that he now dwelt in the Paran Desert.

If Yishmael was no longer in Avraham's house, how did Avraham take him with him to go sacrifice Yitzchok?