Your quote seems to refer to something that Avraham and Yishmael did together. The scope for that is limited.
Avraham was living in Cana'an (Bereishit 13:12) from the time Yismael was born (Bereishit 16) to when he was expelled (Bereishit 21) except for Avraham and Sarah's journey in Bereishit 20. The torah doesn't tell us whether Hagar and Yismael were with them when they made this trip, but for the sake of this question let's assume they were. The question, then, is the area covered by:
And Abraham journeyed from thence toward the land of the South, and dwelt between Kadesh and Shur; and he sojourned in Gerar.
According to Easton's Bible Dictionary as cited by Wikipedia, Gerar is in the Negev desert in Israel. According to the same source, Shur is in the area northeast of Egypt. The location of Kadesh is less clear, in part because it's not clear if Kadesh and Kadesh Barnea are the same place, but all of the places identified by scholars are well north of Mecca, being in the Wildenress of Zin or the Wilderness of Paran.
While it is theoretically possible that, in travelling from Cana'an to Shur, Avraham and his household took a very roundabout detour, the torah gives no reason to think so.
According to the torah narrative, the next time Yishmael and Avraham are together is when Yishmael and Yitzchak come to bury Avraham.
It is, of course, hard to prove a negative. I don't know of any traditions that place them together in that location, but it's always possible there are sources I don't know.