There are different Proofs in the ת"נח for the fact that בית דיו isn't limited to the "חייב OR פוטר" in The חמישי חומשי תורה . )
Society can't function like that, (unless the best Majority of people don't sin) .....Because if that were true if you see someone kill and the witnesses were well known Zadikim, but they're brothers, we must let the Murder go free....
As robev points out in the comments the incident with the meraglim is a case where an entire generation was punished מדה כנגד מדה.
בְּמִסְפַּ֨ר הַיָּמִ֜ים אֲשֶׁר־תַּרְתֶּ֣ם אֶת־הָאָרֶץ֮ אַרְבָּעִ֣ים
יוֹם֒ י֣וֹם לַשָּׁנָ֞ה י֣וֹם לַשָּׁנָ֗ה תִּשְׂאוּ֙ אֶת־עֲוֺנֹ֣תֵיכֶ֔ם
אַרְבָּעִ֖ים שָׁנָ֑ה וִֽידַעְתֶּ֖ם אֶת־תְּנוּאָתִֽי׃