The Mishna in Middos 1:4 refers to a room underneath the Courtyard (Ezras Yisrael), with an entrance to the Women's Courtyard, called Lishkas Pinchas HaMalbish, the Room of Pinchas the Dresser, where all of the priests' priestly garments were stored. It is possible that the High Priest had a separate storage place for his own garments, but this is not specified anywhere.
The specific (two sets of) linen garments used during the Yom Kippur temple service, which Leviticus 16:23 is talking about, were not reused, even from the morning service to the afternoon service. This is understood from the words "והניחם שם", And he shall leave them there. (Rashi to v.23). His regular garments in v.24 refer to the normal set of 8 garments that the High Priest normally wore during temple service.
It is important to note that traditionally (Yoma 32a, cited by Rashi), this verse is understood to be somewhat cryptic and slightly out of place, as it describes an event that occurs after v.24-25:
"[After performing his burnt offering etc. (v.24) and the sin offering
(v.25)] Aaron shall come [again] to the Tent of Meeting* [to remove the
pan of incense]. [Then] he shall take off the linen garments he [had]
put on when he entered the Most Holy Place, and he shall leave them
there."
* In the First and Second Temples, the "Tent of the Meeting" was replaced by the Sanctuary (Heichal) and the Inner Sanctuary (Kodshei Kodoshim).