The gemara brings in Chullin 24b an idea that a kohen is disqualified by blemishes but not by age and a Levi is the opposite. The gemara brings a berisa is in Chullin 24b which states a kohen can serve once he brings 2 hairs until he gets old. Then there is the opnion of Rebbi which says from age 20. The gemara then brings R' Elazar brings from the word "man" that minor kohen is disqualified. The gemara explains that once he brought 2 hairs he can do avodah,but the older Kohanim didmt let until age 20.
So we can say that this is like Rebbi ,and it's just a minhag not to let until 20,or we can say that Rebbi doesnt allow until age 20,and R 'Elazar doesnt allow lechatchila but only bedieved.(There is more info in the gemara which I left out since it was too much info ).
Halacha l'maaseh The Rambam Klei Mikdash 5:15 (which you brought) says that mi'ikar hadin 2 hairs is enough ,but the custom is age 20. This is for a regular kohen. However, it seems that even for a kohen Gadol from age 20 and up is fine,but a kohen gadol has other requirements which they need to fulfill which could invalidate(lchatchila) them if they do not possess them.(see the beg halachos of the perek mentioned above )
On a side note : I did some searching,and found that Aristobulus was appointed as a kohen gadol at age 17,however this should not be a proof simce this was against the Pharisees wishes.