When Prophet Moses went up to the mountain, did all the people end up worshiping the cow, or were there some who remained aloof? What happened to such people who did not become idolaters? Were they also punished?
According to Exodus 32:
The question is a good one, because it doesn't seem perfectly clear from the above verses whether the Levites never sinned or if they quickly realized their error and aligned themselves with G-d and Moses when Moses issued his call.
One possible inference is that, since only about 3,000 people were slain, that means only about 3,000 people sinned. According to the Malbim (cited here), only a small number (the 3,000?) of Bnei Yisrael actually sinned, but the majority of the sinners were killed by plague (See Ex. 32:35), and those were members of the 'Erev Rav, the bandwagon jumpers, so to speak, who had joined Bnei Yisrael when they saw what was going on around them in Egypt during the miracles of the Exodus.
Furthermore, traditionally we understand the story as being something that the men engaged in, not the women, and in fact the men took the jewelry that the women were wearing, against their wishes, and threw it into the fire to make the calf.
It doesn't seem like absolutely everyone sinned, but we have been punished collectively, on different levels, throughout history, in connection with this incident. It does seem, then, that some level of guilt is carried by the Jewish People as a whole.