Is it worse to be a heretic who denies the authenticity of the Mesora, or a follower of Avoda Zarah. are their any rabbinic sources which discuss this question?
See the Gemara Shabbos 116a
Rebbe Tarfon says that in a case where a Rodef is running to kill you, or a snake is chasing to bite you, and in order to escape you could either run and enter into a House of Idol Worship, or a House of Heretics, its better to run into the House of Idol Worship, for the Heretics know the truth and deny it, but the Idol Worshippers deny unknowingly. The Rambam in Hilchos Teffilin 1:13 writes if a min writes a Sefer Torah Teffilin or Mezuza, we burn it, but if nochri or a mumar writes it we bury it since its posul
It would seem to be from this Gemara that Heresy is inherently worse, although one could argue that this is only the case when the idol worshipers are uknowledgeable and are doing it because of "minhag avoseihem b'yedeihem". Nonetheless, we have a clear Gemarah that in a situation of Pikuach Nefesh where we transgress all mitzvos except for the three aveiros, we still pick Avoda Zara over Heresy.