The Lubavitcher Rebbe (Likkutey Sichos vol. 15 pg. 192 and footnote 12-13 there) asserts that (unlike Eisav who the Gemora calls an apostate Jew), Yishmael did not have the legal status of a Jew. He cites the view of many commentators, that only "in Yitzchok will be called your seed" (Genesis 21:12), whereas Yishmael was not considered the seed of Avraham and was disinherited (verse 10 there: "Drive out this handmaid and her son, for the son of this handmaid shall not inherit with my son, with Isaac"). For further elaboration he references to a lengthier discussion on the topic of whether Yishmael was considered Jewish in "Beis Ha'otzar" (Rabbi Yosef Engel, Klal Aleph Ois Gimmel and Ois Yud Ches).
Two verses later, God said to Abraham, "Be not displeased concerning the lad and concerning your handmaid; whatever Sarah tells you, hearken to her voice, for in Isaac will be called your seed."