The gemarra in Shabbos 146a teaches
שבא נחש על חוה הטיל בה זוהמא ישראל שעמדו על הר סיני פסקה זוהמתן
The snake came upon Eve, i.e., when it seduced her to eat from the Tree of Knowledge, it infected her with moral contamination. When the Jewish people stood at Mount Sinai, their contamination ceased
However, I have heard the idea that when the Jews sinned with the Golden Calf, this contamination came back. The earliest source I could find for this was Nefesh HaChaim 1:6:
וע''י חטא העגל חזרה אותה הזוהמא ונתערבה בתוכם כבתחלה . וזש''ה (הושע ו׳:ז׳) והמה כאדם עברו ברית
What's the source for his statement (I don't mean the verse he brings)?