When I was growing up, it was the big three: Shabbos, Kashrus and Taharas haMishpachah. But how could anyone know about the third, anyway?
I don't think there is a source for including kashrus or taharas hamishpachah. I think they are "just" included because they show a level of commitment, a willingness to shape how one lives, even curtail one's basic physical drives, around halakhah.
As for Shabbos, that is well established. At least, public Shabbos observance. The gemara rules on Chullin 5a:
מקבלין קרבנות מפושעי ישראל כדי שיחזרו בהן בתשובה חוץ מן המומר ומנסך את היין ומחלל שבתות בפרהסיא
We accept sacrifices from Jews who are sinners so that they will thereby return in teshuvah -- except from someone entirely non-observant, one who anoints with wine [idolatrously] or who desecrates Shabbos in public.
The Rambam explains the mishnah in Chullin 1:2:
ואמרו "הכל שוחטין", ואפילו ישראל משומד, ויש לכך תנאים ואז תהיה שחיטתו מותרת, האחד שלא יהא עובד עבודה זרה, לפי שאמרו משומד לעבודה זרה משומד לכל התורה כולה. והשני שלא יהא מחלל שבת בפרהסיא לפי שהכלל אצלינו מחלל שבת בפרהסיא הרי הוא כגוי לכל דבריו. ושלא יהיה מין.
And they[, the tannaim, further say in the mishnah], "all may slaughter": Even an intentionally non-observant Jew, although there are preconditions to be met, and then [the meat] of his slaughter is permissible. (1) That he does not worship avodah zarah (polytheism, or worshiping G-d but in a pagan manner. Because they said that someone who is non-observant with regard to avodah zarah is an non-observant with regard to the entire Torah. (2) That he does not desecrate Shabbos in public. Because there is a principle with us that one who desecrates Torah in public is like a non-Jew in all his matter. And [3] that he not be a sectarian...
Just to give a similar sample from the Ashkenazi side, Tosafos (BB 119a, "שנאמר מחלליה מות יומת ...") quote the above gemara and explains it as:
דדומה שבת לע"ז דהיינו מחלל שבת בפרהסיא לעובד ע"ז
... for Shabbos is comparable to avodah zara; that is -- one who desecrates Shabbos in public to one who worships avodah zarah.
Notice the "desecrates Shabbos in public". There has to be a lack of shame about it. (For similar reasons, we aren't talking about someone raised in a non-observant home.) Also, as we homiletically say about the wicked son of the seder "because he removed himself from the community, he denied the fundamental" of our faith. Judaism is inherently communal, and so to be really outside the camp would require violating that camp.
The reason why Shabbos is straightforward. Aside from being life-style changing and thus an indicator of commitment, as are kashrus and taharas hamishpachah, the Torah describes Shabbos as a sign of creation (commandment 5, Shemos 20:11) and a reminder of the Exodus (commandment 5, Devarim 5:11). Shabbos observance thus (see Rashi, Chullin 5a "ela lav H"Q") denying Shabbos and thus that Hashem is the G-d of Creation and of History is akin to altogether worshiping something other than Him.
That said, I think that the ancient litmus test for who is a Jew in good standing was quite different.
Look how the Torah describes Shavous, the anniversary of the giving of the Torah. Vayiqra 23:15,22:
וספרתם לכם ממחרת השבת מיום הביאכם את עמר התנופה שבע שבתות תמימת תהיינה ... ובקצרכם את קציר ארצכם לא תכלה פאת שדך בקצרך ולקט קצירך לא תלקט לעני ולגר תעזב אתם אני ה' אלקיכם
And you shall count for yourselves from the day after the day of rest [ie the first day of Pesach], from the day you bring the omer wave offering, 7 complete weeks it shall be... And when you cut the crops of your land, do not finish off the corners [pei'ah] of your field when you harvest, and the gleanings [leqet] of your harvest, do not collect; leave them for the poor and the stranger -- I am Hashem your G-d!
Which ties into the reading of the narrative of Ruth on Shavuos. Also during harvest time, and also centering on the mitzvos of sharing one's crops with the poor.
For that matter, the gemara Yevamos 47a (and similarly Rambam, Issurei Bi’ah 14:2) explains what we must teach a prospective convert:
ומודיעין אותו מקצת מצות קלות ומקצת מצות חמורות, ומודיעין אותו עון לקט שכחה ופאה ומעשר עני. ומודיעין אותו ענשן של מצות, … וכשם שמודיעין אותו ענשן של מצות, כך מודיעין אותו מתן שכרן… לא רוב טובה ולא רוב פורענות. ואין מרבין עליו, ואין מדקדקין עליו.
And we inform him about some minor mitzvos and some stringent ones, we inform him about the sins of [neglecting] leqet, shikhechah -- one or two forgotten when picking, pei’ah and maaser ani — the tithe to the poor.
And we tell him the punishments of [violating] mitzvos… And just as we tell him the punishments of mitzvos, we tell them the giving of the reward… Not overly about the good, nor overly about the suffering. And we do not overwhelm him, and we do not become unduly meticulous with him.
In sum, we give the prospective convert a "30,000 foot level picture" of what observance means with the exception of leqet, shikhechah, pei'ah and maaser ani. These are the important mitzvos we must make sure every convert knows about.
While we today look at two or three mitzvos between ourselves and G-d as the bellwether mitzvos, that was not true of the Judaism in the days of Tanakh thought at least Chazal's day, if not significantly later. (The Rambam is not a data point, since he is basing himself on the gemara.)
Originally, the definition of an "observant Jew" was measured by the mitzvos that impact how one makes their living.
(I realize that everything below the line is tangential. But anyone aware of current events at the time of this writing will understand why this contrast is a point that needs making. And it does touch on the question of the source, since it argues against the age of the idea and thus the age of possible sources.)