In Bereshit 22:3, Rashi tells us that when Avraham went to sacrifice Yitzchok, the two lads he took with him were Yishmael and Eliezer.
Wasn't Yishmael already gone from Avraham's house? In Bereshit 21:21 we are told that he now dwelt in the Paran Desert.
If Yishmael was no longer in Avraham's house, how did Avraham take him with him to go sacrifice Yitzchok?
Pirkei D'Rebbi Eliezer (Chapter 31) says that Yishmael came in from the desert to visit Avraham. (The Rada"l says that was when Yitzchok and Yishmael argued about whose circumcision was more meritorious - see here) and that night, according to one explanation of the Rada"l, G-d appeared to Avraham and told him to sacrifice his son. (you can see the argument and Satan's discussion with G-d that led to the Akeida in the Sefer Hayashar, brought in this answer)
In Sefer HaYoshor, 22:3, it says: "Then did dwell Ishmael and his sons with Abraham many days." In verse 41, it says, "Then it was when Isaac was thirty-years-old and seven years and Ishmael his brother was walking and he had come with him into the tent." The story of Isaac's binding is in the following chapter. Therefore, we can see that Ishmael was living at the time, with Abraham.
