When Prophet Moses went up to the mountain, did all the people end up worshiping the cow, or were there some who remained aloof? What happened to such people who did not become idolaters? Were they also punished?
According to Exodus 32:
The question is a good one, because it doesn't seem perfectly clear from the above verses whether the Levites never sinned or if they quickly realized their error and aligned themselves with G-d and Moses when Moses issued his call.
One possible inference is that, since only about 3,000 people were slain, that means only about 3,000 people sinned. According to the Malbim (cited here), only a small number (the 3,000?) of Bnei Yisrael actually sinned, but the majority of the sinners were killed by plague (See Ex. 32:35), and those were members of the 'Erev Rav, the bandwagon jumpers, so to speak, who had joined Bnei Yisrael when they saw what was going on around them in Egypt during the miracles of the Exodus.
Furthermore, traditionally we understand the story as being something that the men engaged in, not the women, and in fact the men took the jewelry that the women were wearing, against their wishes, and threw it into the fire to make the calf.
It doesn't seem like absolutely everyone sinned, but we have been punished collectively, on different levels, throughout history, in connection with this incident. It does seem, then, that some level of guilt is carried by the Jewish People as a whole.
Rashi says in his commentary on Exodus 32:20 there were different punishments for different levels of sin. People who were warned before witnesses were killed by the sword, as was done by an Idolatrous city, (Deut. 13:16). Others who acted before witnesses without warning were killed in a plague, (Ex. 32:35). Others who acted without witness or warning were given the water with the calf's ashes to drink, and had their stomachs burst if they were guilty, like the sotah, (Num. 5:27).
Based on this, all manner of sin here resulted in death. Since our nation is still extant, it seems evident that we did not all worship the golden calf.