Part of the reason has to do with the deeper essence of what happens during sex.
The male sperm is not just physical material, it carries with it part of the man's soul. If used for its proper purpose, it brings life, but if not, it brings death to the world. (see more details here)
there are many sources for this, for example the zohar says when Adam separated from his wife after the sin, he had seminal emissions which created all kinds of destructive forces.
Likewise Rashi writes in parashas Noach (Bereishis 6:13): "Every place that one finds sexual immorality androlemusia comes to the world killing both the righteous and the evil."
On a personal level, sexual immorality (such as masturbation) brings tremendous sadness and this is due to these destructive forces which attach to a person who commits these things. The more severe the immorality the more powerful the destructive force and the more sadness "sticks" to the person. This is supported by the much higher suicide rates among gays than straight men. (though one can claim this is due to other factors)
Hence, much of the reason some people don't think it's a big deal is because of ignorance as to what happens during sex.
There are also practical reasons besides this such as the general moral decadence it brings to oneself and society and therefore God calls it an abomination.