Did all of Bnei Israel worship the golden calf?

Question

When Prophet Moses went up to the mountain, did all the people end up worshiping the cow, or were there some who remained aloof? What happened to such people who did not become idolaters? Were they also punished?

Answer 1

According to Exodus 32:

19 And it came to pass, as soon as he came nigh unto the camp, that he saw the calf and the dancing; and Moses' anger waxed hot, and he cast the tables out of his hands, and broke them beneath the mount.
20 And he took the calf which they had made, and burnt it with fire, and ground it to powder, and strewed it upon the water, and made the children of Israel drink of it.
21 And Moses said unto Aaron: 'What did this people unto thee, that thou hast brought a great sin upon them?'
22 And Aaron said: 'Let not the anger of my lord wax hot; thou knowest the people, that they are set on evil.
23 So they said unto me: Make us a god, which shall go before us; for as for this Moses, the man that brought us up out of the land of Egypt, we know not what is become of him.
24 And I said unto them: Whosoever hath any gold, let them break it off; so they gave it me; and I cast it into the fire, and there came out this calf.'
25 And when Moses saw that the people were broken loose--for Aaron had let them loose for a derision among their enemies--
26 then Moses stood in the gate of the camp, and said: 'Whoso is on the LORD'S side, let him come unto me.' And all the sons of Levi gathered themselves together unto him.
27 And he said unto them: 'Thus saith the LORD, the God of Israel: Put ye every man his sword upon his thigh, and go to and fro from gate to gate throughout the camp, and slay every man his brother, and every man his companion, and every man his neighbour.'
28 And the sons of Levi did according to the word of Moses; and there fell of the people that day about three thousand men.

The question is a good one, because it doesn't seem perfectly clear from the above verses whether the Levites never sinned or if they quickly realized their error and aligned themselves with G-d and Moses when Moses issued his call.

One possible inference is that, since only about 3,000 people were slain, that means only about 3,000 people sinned. According to the Malbim (cited here), only a small number (the 3,000?) of Bnei Yisrael actually sinned, but the majority of the sinners were killed by plague (See Ex. 32:35), and those were members of the 'Erev Rav, the bandwagon jumpers, so to speak, who had joined Bnei Yisrael when they saw what was going on around them in Egypt during the miracles of the Exodus.

Furthermore, traditionally we understand the story as being something that the men engaged in, not the women, and in fact the men took the jewelry that the women were wearing, against their wishes, and threw it into the fire to make the calf.

It doesn't seem like absolutely everyone sinned, but we have been punished collectively, on different levels, throughout history, in connection with this incident. It does seem, then, that some level of guilt is carried by the Jewish People as a whole.