Was Hagar from Shem, Cham, or Yefes? If we say she was from Cham, then how can this be understood in light of Avraham's going out of his way to make sure his descendants don't marry into the descendants of Cham? According to Rashi, he even took her back after Sarah's death.
The Chida in Sefer Devash L'Pi 1:24 says that Hagar came from Cham.
Regarding your question "how can this be understood in light of Avraham's going out of his way to make sure his descendants don't marry into the descendants of Cham?" A look at the verses in Braishis does not mention Cham, it only mentions Canaan. (Breishis 24:3). Rashi explains that since Canaan was cursed, one that is blessed does not stick with one that is cursed. Looking in Parshas Noach 9:25 only Canaan was cursed, not all of Cham's offspring. Hagar although she was from Cham, most likely was not from the offspring of Canaan, more likely she was from the offspring of Mitzrayim, a different son of Cham who was not cursed.
According to the following Hebrew article Hagar was Cham's daughter: