Shouldn't the obligation to sit in the Sukkah also fall into the category of Sefeika D'yoma? If so why do some not eat in the Sukkah on Shemini Atzeres?
To answer the first question: yes. Because of 'sfeika deyoma', one should sit in the sukkah even on Shemini Atzeres just as one would on the other days of Sukkos. The Gemara Sukkah 47a (as we have it) concludes this way, and says that we should sit in the Sukkah, but not make a bracha. (The reasons for this, as I understand it, is either because it might be a bracha levatalah, see Rosh 4:5, or because we need to differentiate it from Sukkos proper, to avoid bal tosif, which I believe is the opinion of the Raavya).
However, as your second question notes, this is not - and was not - always the prevailing practice. There are two reasons for this al pi halakha (as opposed to kabbalah):
Some of the Rishonim may have had a different version in their text of the Gemara, which lacks the conclusion that we should sit in the sukkah on Shemini Atzeres. See Sefer Yechusei Tanaim Ve'amoraim, printed here, who writes that according to the usual rules of how to pasken from the Gemara, we should not sit in the sukkah on the 8th day, especially because we already begin to ask for rain on Shemini Atzeres (see Taanis 4b). Thus, he reasons, the Gemara paskens that sitting in the Sukkah when it isn't actually sukkos would be 'adding on to the mitzvah' inappropriately; since a person will be declaring in kiddush that the day is "hashmini", he's implying that there's an obligation to sit in the sukkah on the 8th day. (This is the gist of R. Tzadok's justification as well, besides for many others)
The Aruch Hashulchan (O.C. 668:4, probably based on the Korban Nesanel to the Rosh 4:5) justifies the custom, despite believing that the Gemara actually states that one is obligated to sit in the sukkah on shemini atzeres. He writes that the Gemara was referring only to places where the weather is nice enough that people would be sitting outside anyways, as Tosfos (there, Sukkah 47a) write, "the sukkah is pleasant to him", and so sitting in the sukka isn't an obvious slight to the Yom Tov of Shemini Atzeres. However, in the European countries, the weather is so cold that one might argue that we are always exempt from living in the sukkah, though we sit there anyway because our desire to fulfill the mitzvah overcomes this discomfort. However, doing something specifically for a mitzvah of sukkos should not be done on Shmini Atzeres because of zilzul yom tov (disgracing the holiday, by indicating that it might be chol hamoed) and since sitting in a sukkah would obviously be for the purposes of a mitzvah, we shouldn't do so on yom tov, just like we don't shake the lulav. Many poskim give similar reasons (such as Minchas Elazar 4:31)
The simple answer would be that we avoid any behavior that slaps the new yomtov of Shmini Atzeres in the face (such as shaking lulav). To eat in the sukkah is no big deal, as plenty of people eat on porches or gazebos or what-have-you when the weather's nice, no religious ritual required. So in places where it was ridiculously cold on Shmini Atzeres, they didn't!
There are Hassidic thinkers who have answers that give the practice more significance (often based on Hassidic or kabbalistic thought), but that would be the simplest answer.