There are two levels of Divine justice that may affect sinners.
1) Punishment for the crime itself.
2) Extra-judicial punishment in the special case called Chillul Hashem.
Both of these are considered true justice. But, Chillul Hashem (desecrating G-d's name) is applied to the sin AND the public situation caused, even if the sinner(s) do not deserve the full measure of justice meted out if we only looked at the sin by itself.
For instance, Moshe Rabbeinu was punished with dying outside of Eretz Yisrael, because he did not sanctify Hashem in front of the people by the waters of Merivah. Aharon shared the same fate simply because he didn't protest when Moshe hit the rock. See Bamidbar 20: 12-14 with Rashi, who explains that when Hashem punishes his holy ones, the creations fear Him. (Therefore we see that the punishment of those who cause Chillul Hashem restores the fear of Hashem in the world.)
Obviously, the sentence of premature death and being barred from the land, was not in line with the crime committed. Normally we would say that such a sentence is much too severe for the crime.
However, where a public desecration of G-d's name results, the sinner needs to be dealt capital punishment as a matter of justice, because that is the way the name of heaven's honor, is restored. This does not mean that the people involved really deserve to be killed. It means that the situation calls for their deaths to repair the breach which is their fault. Their death sends a message that such desecration cannot be tolerated.
You can know this is true because nowhere do we say Moshe and Aharon died as sinners. They are righteous people who made a mistake. (see Rashi Bamidbar 20:12) It is just that the mistake, when considering their closeness to Hashem (high level), and the resulting loss of respect to Hashem caused in public, demands repair.
Rashi explains: (Bamidbar 11:22) that R' Akiva asks why Moshe Rabeinu was not punished for saying that Hashem could not provide meat? Was such disbelief worse than hitting the rock? He answers that since the statement about meat was said in private, he was not punished; whereas the waters of Merivah was a public desecration.
Now we can understand what happened here in the OP's case by the golden calf.
The entire Jewish people had just achieved the level of mass prophecy in which they heard Hashem say "Anochi" and "You shall not have other gods before me" etc. A few days later, some of the people publicly worshipped a golden calf. By the justice of the crime itself, the people who did not actually worship the calf should be innocent of the death penalty. However, since they did not protest, the public desecration of Hashem's name made them all culpable to destruction.
However, Moshe Rabbeinu knew that the punishment was not because the sin of failing to protest deserves death itself, but rather that the situation called for their deaths to repair the breach of honor - "Chillul Hashem".
In a move of genius, Moshe Rabbeinu brings the defense of "Egypt will say..". Moshe's reasoning is "Mimah nafshach" either way the Name is desecrated. In this particular case, killing the offenders will not serve the intended purpose of restoring the fear of heaven in the world, because if you kill them, it will just replace the desecration of the Name with another equal desecration! ("Egypt will say..") So therefore, it would be unjust to punish sinners for the mere crime of not protesting, with the death penalty, since their death accomplishes nothing towards justice!
Hashem answered: OK, true, but you Moshe also left Egypt, and I can make you into the new nation(Shmos 32:10). Therefore, the sinners can die to repair the breach, and you will carry on so Egypt will not be able to fully say that all the people died in the desert? "CHECK"
Moshe answers: Shmos 32:32 "Then erase me from Your book." I want no part of it. Now You are back to Mimah nafshach. Either way the Name is desecrated. "CHECKMATE"
Hashem: ...NICE!
However, that only reduced the situation back to regular justice. So now, the usual punishments to the various sinners needed to be meted out. So Moshe has to "clean up the camp".
