Excellent question! Avot D'Rav Natan ch. 9 Mishna 2 explains why. In summary, he states that that B'nai Yisra'el were tested 10 times (The Mishnah lists all 10) but they weren't punished for any of them except for the incident with the spies, because this was Lashon Hara.
At the end of the Mishnah, he states, that this is a kal vachomer we should learn, that if G-d punished for Lashon Hara about a parcel of land that has no mouth to speak to defend itself, how much more so must we be careful about speaking Lashon Hara against another person that can speak!
I would take this idea a bit further - G-d promised our forefathers not only that B'nai Yisra'el would inherit the land of Israel as an eternal inheritance, but He made sure that this land was a beautiful land flowing with milk and honey. Furthermore, the original idea was that G-d was to banish all the inhabitants for us, so that we could just come and live there without having to fight any wars to gain that land. In short, the land was a beautiful gift. And, B'nai Yisra'el dared to speak Lashon Hara about that land. Essentially, they "bad-mouthed" a gift, and the essence and very "reason" of their own existence! That sounds like not just a crime worthy of serious punishment, but a self-inflicted punishment! Essentially they rejected the gift given to them in a personal way to the ONE that gave them the gift. Thus, they weren't worthy of having that gift.