The Chizkuni, in his commentary on pasuk 28 writes that these 3.000 people needed to be killed because these 3.000 men intended that their conduct should be viewed as idolatrous:
כשלשת אלפי איש, “approximately three thousand men.” According to the plain meaning of the text, these three thousand men intended that their conduct should be viewed as idolatrous. This is why they had to be executed, just as the inhabitants of the idolatrous town in Deuteronomy chapter 14 have to be executed and their belongings have to be burned. (Based on Daat Baaley Tossaphot)
The Pirkei DeRabbi Eliezer (chapter 45:11) explains something similair to what the Chizkuni writes, e.g. that these people intended that their conduct should be viewed as idolatrous. The Pirkei DeRabbi Eliezer explains that all the people who kissed the golden calf with all his heart, his upper lip and his bones became golden:
He made Israel drink the water (with the dust of the calf). Everyone who had kissed the calf with all his heart, his upper lip and his bones became golden, and the tribe of Levi slew him, until there fell of Israel about three thousand men, as it is said, "And the sons of Levi did according to the word of Moses" (Ex. 32:28).
Note that the Pirkei DeRabbi Eliezer mentions the words "with all his heart", it seems to me that this supports the view of the Chizkuni, that their hearts were filled with the intention of worshipping the golden calf.
A really interesting approach to this is given by the Shelah in his commentary on this pasuk. He mentions that they (the 3.000 men) were already sentenced to death when they leaved Mitzrayim, however, their sentence was suspended temporarily. The episode of the sin with the golden calf was however when the sentence was fulfilled (Shenei Luchot HaBerit, Torah Shebikhtav, Ki Tisa, Torah Ohr):
Presumably, their guilt had been that they had already then harboured thoughts of an idolatrous nature, but had only found an outlet for these thoughts during the episode of the golden calf.
The Shelah here mentions the
"thoughts of an idolatrous nature", as does the Chizkuni.