Is it worse to be a heretic who denies the authenticity of the Mesora, or a follower of Avoda Zarah. are their any rabbinic sources which discuss this question?
See the Gemara Shabbos 116a
Rebbe Tarfon says that in a case where a Rodef is running to kill you, or a snake is chasing to bite you, and in order to escape you could either run and enter into a House of Idol Worship, or a House of Heretics, its better to run into the House of Idol Worship, for the Heretics know the truth and deny it, but the Idol Worshippers deny unknowingly. The Rambam in Hilchos Teffilin 1:13 writes if a min writes a Sefer Torah Teffilin or Mezuza, we burn it, but if nochri or a mumar writes it we bury it since its posul
It would seem to be from this Gemara that Heresy is inherently worse, although one could argue that this is only the case when the idol worshipers are uknowledgeable and are doing it because of "minhag avoseihem b'yedeihem". Nonetheless, we have a clear Gemarah that in a situation of Pikuach Nefesh where we transgress all mitzvos except for the three aveiros, we still pick Avoda Zara over Heresy.
Rambam discusses this question in the Guide for the Perplexed 1:30 and clearly states that heretics are worse than idol-worshippers (and not for the reason mentioned by the Gemara quoted in the answer given by @Shoel U'Meishiv ).
How great, then, must be the offence of him who has a wrong opinion of God Himself, and believes Him to be different from what He truly is, i.e., assumes that He does not exist, that He consists of two elements, that He is corporeal, that He is subject to external influence, or ascribes to Him any defect whatever. Such a person is undoubtedly worse than he who worships idols in the belief that they, as agents, can do good or evil.
Therefore bear in mind that by the belief in the corporeality or in anything connected with corporeality, you would provoke God to jealousy and wrath, kindle His fire and anger, become His foe, His enemy, and His adversary in a higher degree than by the worship of idols. If you think that there is an excuse for those who believe in the corporeality of God on the ground of their training, their ignorance or their defective comprehension, you must make the same concession to the worshippers of idols: their worship is due to ignorance, or to early training, "they continue in the custom of their fathers." (TḄ. Ḥullin, 13a) You will perhaps say that the literal interpretation of the Bible causes men to fall into that doubt, but you must know that idolaters were likewise brought to their belief by false imaginations and ideas. There is no excuse whatever for those who, being unable to think for themselves, do not accept [the doctrine of the incorporeality of God] from the true philosophers. I do not consider those men as infidels who are unable to prove the incorporeality, but I hold those to be so Who do not believe it, especially when they see that Onkelos and Jonathan avoid [in reference to God] expressions implying corporeality as much as possible. (Friedlander translation, my emphasis.)