The mishnah in Sukkah (48b) tells of a Tzadukki who, on Sukkot, performed a service in the temple incorrectly and all the people in the temple stoned him to death with their esrogim. I would like to know how they were halachically allowed to do so instead of taking him to court and having the normal proceedings of a halachic capital case.
It's not a capital offense to perform a service in the temple incorrectly. So you don't have to assume that he was killed.
That said, the Mishna in Sanhedrin פרק ט - משנה ו mentions that if a Cohen performed the service while impure - the punishment for which is lashes by Bet Din - the Cohen Youth Movement would remove him from the temple and bash his head in with wooden sticks.
כֹּהֵן שֶׁשִּׁמֵּשׁ בְּטֻמְאָה, אֵין אֶחָיו הַכֹּהֲנִים מְבִיאִין אוֹתוֹ לְבֵית דִּין, אֶלָּא פִרְחֵי כְהֻנָּה מוֹצִיאִין אוֹתוֹ חוּץ לָעֲזָרָה וּמַפְצִיעִין אֶת מֹחוֹ בִּגְזִירִין
So there is a precedent for the enraged to kill somebody who doesn't do the service correctly.
There are actually 3 other cases in that Mishna of people who are killed by the enraged rather than being punished by Bet Din:
הַגּוֹנֵב אֶת הַקַּסְוָה וְהַמְקַלֵּל בַּקּוֹסֵם וְהַבּוֹעֵל אֲרַמִּית, קַנָּאִין פּוֹגְעִין בּוֹ
- One who steals a temple vessel
- One who curses the Lord with the name of an idol
- One who has marital relations with a non-Jewess
So the concept of extra-Bet-Din execution by the enraged is not unheard of in Judaism.
Avodah Zarah 26b:
המינין והמסורות והמומרים היו מורידין ולא מעלין
Heretics, informers, and sinners to anger Hashem - we kill them and do not save them.
As the Kohen showed that he's a Tzeduki, he qualifies as a heretic and therefore may be killed without a court case.