The vast majority of shuls self-identified as Orthodox are mainstream. So chancechances are, any Orthodox shul or community you approach will be mainstream.
Sabbateans are non-existent, as far as I know.
Karaites are very rare and will not pretend to be Orthodox.
Any Orthodox shul without a mechitza (I hear there are a few left) is outside of the mainstream, at least on that issue.
Messianic (that is, Christian) Jews will sometimes present themselves as practicing legitimate Judaism (though the website of the shul will nearly always reveal the fact that they are Messianic). But if you hear people talking about "Yeshua" being G-d or providing salvation or anything of that nature, you'll know you're dealing with Messianists.
Lubavitcher (Chabad) shuls that are openly meshichist (ie, believe their late Rebbe is Moshiach) are considered outside the mainstream by most Orthodox Jews. Official Chabad houses, listed on Chabad.org for example, are virtually never meshichist, and thus would be considered mainstream by the vast majority of Orthodox Jews (except those few that see contemporary Chabad as illegitimate in general.)
To determine whether a Chabad shul or community is openly meshichist, look at associated websites to see whether there are references to their Rebbe being Moshiach. In addition, attend community events, shul, etc. and see if anyone is wearing yarmulkes with the "yechi" slogan on them. Even if there are some people with such yarmulkes, that does not mean that the entire community is meshichist.
There are a small number of charedi cults. These can be identified in the same way a non-Jewish cult can be identified. For example -- that is, does the rabbi exert an unusually high level of control over individualsby looking for particular cult characteristics researchers have identified, etcor by seeing if the group has been labelled a cult by ex-members or cult experts.?
As a final method of determining whether a community is mainstream, look at the website of the shul, and specifically at the bio of the rabbi. If he got his rabbinic ordination from Yeshivat Chovovei Torah, the shul may be considered "open Orthodox," a philosophy which according to some Orthodox Jews is not truly Orthodox, because of the many ways in which the statements of associated rabbis resemble Conservative Judaism. If
Conversely, if the ordination is from a well-known institution, such as Yeshiva University (Modern/Centrist Orthodoxy), Rabbinical College of America-Lubavitch (Chabad), or Beth Medrash Govoha (Yeshivish), this may be an indication that the community is mainstream. However, since as noted above non-mainstream communities are rare, semicha from a less well-known institution does not mean there is a significant chance of the community being out of the mainstream.