Hashem says: “All who anger me shall not see [the land]” (Bamidbar 14:33)
According to the Midrash (see חידושי מרן הגרי"ז who brings this medrash), this means that everyone, even the young children, would die before entering the land.
From this we can ask a seemingly obvious question: why did everyone need to be punished? The infants did not take part in the sin; what did they do wrong that they should not be allowed to go into Eretz Yisrael?
The griz does not cite his source but it is the medrash rabba on shlach 23. The medrash as brought by the griz darshans that there were different groups to be dealt with. The group you mention were the ones who brought signs of adulthood but were not yet twenty, if they were involved in 'the eitza' they were punished, if they were not involved they were not. Anyone under the age of adulthood was not punished even if they were involved. Anyone above twenty was punished even if not involved. The griz also points out that the טפכם who were the ones guaranteed entry were the ones the medrash calls 'not having brought signs of adulthood' . Seems fine to me.
