When Christianity began many followers considered themselves Jews, the same could be said for followers of Shabbtai Tzvi. I don't believe there is anyone today who would argue that a child born to Christian or Donmeh parents has the status of a Jew. I would like to know if there is any sort of halachic demarcation line for when a sect of Judaism would be considered a different religion such that children born to parents who are part of that sect would not be considered Jewish.
Rav Aharon Lichtenstein zt"l wrote an essay / teshuvah titled "Brother Daniel and the Jewish Fraternity" (Judaism 12:260-280, Summer 1963). Brother Daniel was a Catholic monk who applied for Israeli citizenship under the Law of Return, since he was born Jewish. In it, Rav Lichtenstein deals with this very question.
While we generally repeat the gemara that says that once someone converted, even if they returned to their old faith, they are still Jewish. An apostate Jew is still a Jew, and his marriage valid. (Yevamos 47b) As the idiom goes, "ישראל, אף על פי שחטא, ישראל הוא -- an Israelite, even though he sinned, is an Israelite."
However, there are two other gemaras. R' Assi worries about the potential validity of a marriage between a non-Jew and a Jew, because it is possible that the alleged non-Jew is actually a descendent of one of the 10 lost tribes. (Yevamos 16b) But the gemara concludes (Yevamos 17a) with Shmuel's commitment to find a disproof. Apparently the 10 tribes are no longer "an Israelite, even though he sinned." Quoting Hosea, one of their last prophets who warned them (Hosea 5:7)
"בַּה' בָּגָדוּ, כִּי-בָנִים זָרִים יָלָדוּ; עַתָּה יֹאכְלֵם חֹדֶשׁ, אֶת-חֶלְקֵיהֶם
They were treacherous with Hashem, for they have begotten strange children, now shall the new moon consume them [and?] their portion."
R' Aharon cites a second gemara with this position. Rav Ami and Rav Assi (Chullin 6a) learned that the Kusim (Samaritans) were not observing the Torah. "זזו משם עד שעשאום עובדי כוכבים גמורין - and they did not move from there until they made them [the Kusim] full-fledged gentiles.
So how is it that a backslid convert is considered still Jewish, but the 10 Lost Tribes and Samaritans are not?
Rav Aharon lists three possible resolutions:
1) The Rashba tones down the statement about the Kusim. He says yes, they are indeed Jewish halachically, but since the rabbinate has the power to make or break marriages, they invalidated the marriages of Samaritans, making the marriage invalid LIKE those between a Jew and a non-Jew. Similarly, the Shulchan Arukh (YD 159:3) rules that Kusim are included in the prohibition against charging or paying interest.
2) The Mordechai (Hagahos Mordechai Yavamos 107) takes the opposite approach -- the validity of the marriage of the backslid convert is only as a stringency. We cannot invalidate the marriage because the husband might have thoughts of teshuvah. But if we could read minds and know that he really did return to his old faith, his Judaism would evaporate.
(I am pausing my summary of the article to note that while this is theoretically interesting, I would be hard pressed to believe the Mordechai's position is a viable ruling today for pragmatic use. Consensus seems to have retired it. Perhaps as one factor among others when a rabbi is seeking a way to let a wife remarry when her apostate husband won't give her a get.)
3) Rav Aharon rules like his wife's great-grandfather's, Rav Chaim Brisker's resolution of the apparent contradiction. The Rambam rules that the apostate's marriage is still valid (see Hilchos Ishus 4:15 and Hil' Issurei Bi'ah 13:17). And he writes in his commentary on the mishnah to Niddah 7:4 that something that is in an inclosed with a Kusi's body in it is not tamei, like it would be if it were a Jewish body.
Rav Chaim (and Rav Lichtenstein) understands the Rambam as distinguishing between the cases on the grounds of core identity. The apostate who goes back to acting as he did before converting to Judaism, even in worshipping idols (or the trinity), committed what Rav Aharon calls an "apostasy of action". However, the 10 Tribes and the Kusim lost their core identity. When the person gives up their Jewish roots, Jewishness is indeed lost.
I think that gives two or maybe three valid answers to your question. (Depending if you agree to what I wrote about consensus burying the viability of ruling like the Mordekhai today.)
However, Rav Aharon Lichtenstein rules in practice like the third. Brother Daniel joined another faith community. He doesn't try to be one of the Jewish Community. So, he is not halachically Jewish.
Meanwhile, Brother Daniel may share more beliefs with Rav Aharon than would the typical Reform Rabbi -- he believes that the Torah was given to Moshe (perhaps even dictated) in the Sinai, that the Exodus really happened as told, that G-d "Spoke" to the Jews at Sinai, etc... But the Reform Rabbi did not give up his Jewish roots.
The apostate Jew who sees himself as just that -- an differently-believing Jew, and a member of the Jewish community -- is not obligated to convert. However, lechatchila (if we can, before the fact) we would require a dunk in the mikvah as a sign of commitment before counting him toward a minyan.
A baal teshuvah doesn't need to go to the mikvah before we count him toward a minyan, because typically they are not apostates in the active sense, they are products of upbringing (tinoqos shenishbu) who happened to end up believing apostasy.
But Brother Daniel, because he left the Jewish People except when there was a legal advantage to pretend otherwise, would actually need that dunking halachically, at least according to this ruling.
Dr. Lawrence Schiffman addresses this question in his book Who Was a Jew?: Rabbinic and Halakhic Perspectives on the Jewish Christian Schism. The answer he provides for Christianity is that it was a gradual process but there were major milestones such as the promulgation of the blessing against heretics in the amida and the banning of Jews from Jerusalem after the Bar Kochva revolt. These factors combined to make it abundantly clear to the outside world who was a Jew and who was a Christian. Jewish Christians may have persisted in smaller groups after that and a possible final demarcation would be the declaration of Christianity as the official religion of the Roman Empire.
I would speculate that similar world events helped to identify that followers of Shabbtai Tzvi were no longer considered part of Judaism after he apostatized.
A form of worship(or even "sect" of Judaism) halachically becomes distinct from Judaism when it rejects any or all of the fundamentals of Judaism and adopts halachic practices and processes rejected by Chazal. Children born to these sects remain Jewish until they are born to a non-Jewish woman. Once they are borne from a non-Jewish woman, the child loses their status as a Jew and requires conversion, should they wish to become Jewish.
The fundamentals of Judaism have been redacted to 13 principles by the Rambam and can be found here:https://www.chabad.org/library/article_cdo/aid/332555/jewish/Maimonides-13-Principles-of-Faith.htm.
These are them(credit to chabad.org):
Belief in the existence of the Creator, who is perfect in every manner of existence and is the Primary Cause of all that exists.
The belief in G‑d's absolute and unparalleled unity.
The belief in G‑d's non-corporeality, nor that He will be affected by any physical occurrences, such as movement, or rest, or dwelling.
The belief in G‑d's eternity.
The imperative to worship G‑d exclusively and no foreign false gods.
The belief that G‑d communicates with man through prophecy.
The belief in the primacy of the prophecy of Moses our teacher.
The belief in the divine origin of the Torah.
The belief in the immutability of the Torah.
The belief in G‑d's omniscience and providence.
The belief in divine reward and retribution.
The belief in the arrival of the Messiah and the messianic era.
The belief in the resurrection of the dead.
The early catholics rejected the Torah and replaced it with a "new testament"; thus changing the Torah in violation of principle #9. They also violated principle #5.
The Tzadukim and Beitusim rejected the principle of olam haba, in violation of principles 12 & 13.
The Karaites rejected that the Torah is divine, in violation of principle #8.
The kutim worshipped a small dove, in violation of principle #5.
The Reform and Conservative movements rejected the belief in one God(or don't require it, at least), that the Torah is divine, and more. Although many of them are still Jews(for now), they marry non-Jews freely and their children and/or grandchildren are not or won't be Jewish once they will be born to a non-Jewish woman. See here for a chart based on statistics from the Pew Research center:https://www.simpletoremember.com/grandchildren/WYGBJpdf.pdf.
As with most of the above listed sects, the first generations were Jewish, but they eventually married out(demarcation/halachic line of maintaining Jewish status) and lost their status as Jews. The same will be true and is true of every religion that doesn't adhere to Judaism; they eventually marry out and bear non-Jewish children until the majority/totality of the sect becomes non-Jewish.