I also used to think it had to do with Moshe's ageit had to do with Moshe's age. But then this made me wonder why we say "You should live till 120", since it sounds more like a curse than a blessing to limit someone's potential age to a set number of years. Then it was pointed out to me that it isn't a curse, but a determination made by HaShem, explicitly stated in the Torah.
In BeReishith (6:3) HaShem says: "לֹא-יָדוֹן רוּחִי בָאָדָם לְעֹלָם, בְּשַׁגַּם, הוּא בָשָׂר; וְהָיוּ יָמָיו, מֵאָה וְעֶשְׂרִים שָׁנָה. "
"My spirit shall not abide in man for ever, for that he also is flesh; therefore shall his days be a hundred and twenty years."
Essentially, we are saying that the person should live to the maximum age he can, which is set at 120 years as ordained by HaShem.
Rashi holds that the Pasuk is referring to something elseRashi holds that the Pasuk is referring to something else, but I believe that is likely where the tradition stems from. Rashi's opinion is not shared by everyoneRashi's opinion is not shared by everyone.