During the holiday of sukkot, whenever you eat, sleep or do other "living" type activities they must be done within the sukkah. The exception to this rule is that if you are caused pain because of bad weather conditions or mosquitoes you might be exempt (consult your kitzur/shulchan aruch for the sources).
The reason I say "assur" and not simply "exempt" is because you are not allowed to cause yourself tzar on yom tov.
How cold does it have to get for it to become assur to sit in the sukkah during the holiday of sukkot? Can't you always add on more sweaters/coats?