There are minor pieces subject to tearing (like the techeles strands of the tzitz and the ephod), but those are easily replaced and aren't intrinsic to the remainder of the beged.
Most of the other begadim of the Kohen Gadol contain metal strands or are actually made of metal, making them difficult to "tear" - you would have to cut them. The other begadim also have straps that allow them to be easily loosened and removed.
The me'il is subject potentially to significant wear as the kohen gadol takes his head in and out of the opening without any buttons or ties to loosen the garment. The neck of the me'il is therefore more subject to tearing than any other piece of his begadim.
The sefer hachinuch you linked says as follows: משרשי המצוה לפי שהקריעה דבל
סל גנאי אצלנו ועניין הסתתה ואף כ׳ בפי הבגד נתרחקנו
מן הדבר והיזהרנו עליו בלאו כדי שילבסוהו הלובסו
באימה וביראה ובנחח ובדרך כבוד שיירא מלקורעו ומלהשחית
בו דבר
It makes clear that this is a richuk/azhara made ESPECIALLY because the pi of the me'il is inclined to tear moreso than the other begadim.
It isn't that the other begadim don't need to be treated with kavod, it's just particularly important vis a vis the me'il.