In Sefer HaYoshor, 22:3Sefer HaYoshor, 22:3, it says: "Then did dwell Ishmael and his sons with Abraham many days." In verse 41verse 41, it says, "Then it was when Isaac was thirty-years-old and seven years and Ishmael his brother was walking and he had come with him into the tent." The story of Isaac's binding is in the following chapter. Therefore, we can see that Ishmael was living at the time, with Abraham.