I think this answers it. see Yetzer Enticement #19 of the Chovos Halevavos Gate 5 ch.5 regarding not considering yourself better than a seemingly bad person.
And even if his appearance is bad, it is possible that the reason is because he is ignorant of his obligations to the Creator. Therefore he is more pardonable than me, because my knowledge is greater than his. For the Creator claims from a man only according to the extent of his knowledge. Therefore, I am more deserving to be considered reprehensible for my shortcomings in the Creator's service, despite my knowledge compared, to this man whose shortcoming is due to his ignorance. He rebels against G-d due to ignorance and error, while I rebel against Him knowingly and deliberately.