The three oldest sources that I can find on this question are Shemot Rabbah 7:1, Pirqei deRebi Eliezer 39 and Seder Olam Rabbah 2. They read as follows:
הדא הוא דכתיב בכל עצב יהיה מותר ודבר שפתיים אך למחסור. בכל הדברים שאדם
נושא ונותן בהם דברי תורה הוא נוטל עליהן שכר יכול אף בדברי בטלה כן
תלמוד לומר ודבר שפתיים אך למחסור. אתה מוצא לא היה יוסף ראוי לנתן בבית
האסורין אלא עשר שנים מפני שהוציא דבה על עשרת אחיו ועל ידי שאמר לשר
המשקים כי אם זכרתני אתך והזכרתני אל פרעה נתוסף לו עוד שתי שנים
Here it is written, "From all toil there is some gain, but idle
chatter is pure loss" (Proverbs 14:23, JPS). Should one's business
matters concern words of Torah, he receives reward for them. It is
likewise with words of idleness, as it says: "but idle chatter is pure
loss". So you find that Joseph was only deserving of being placed in
prison for ten years, since he delivered an evil report of his ten
brothers, but due to the fact that he said to the chief cupbearer,
"But think of me... mentioning me to Pharaoh" (Genesis 40:14, JPS), a
further two years were added to his sentence.
והביאה עליו האשה עלילות דברים ונחבש בבית האסורים עשר שנים
And the woman brought false charges against him and he was locked up
in the prison for ten years.
[Rabbi David Luria (19th c. Lithuanian), in his peirush on PRE, suggests that this is a reference to the length of time that Joseph spent in prison before the extra two years were added.]
ירד יוסף למצרים ועשה בבית פוטיפרע י"ב חדשים שנא' ויהי מאז הפקיד אותו
וגו' בבית מפני החמה ובשדה מפני הצינה ועשה בבית האסורים י"ב שנה שנא'
ענו בכבל רגליו וגו' עד עת בא דברו וגומר
Joseph went down to Egypt and worked in Potiphar's house for twelve
months, as it says: "And from the time that [the Egyptian] put him in
charge... [the blessing of the Lord was upon everything that he owned,
in the house and outside]" (Genesis 39:5, JPS). In the house because
of the sun [ie: the summer]; and outside because of the cold [ie: the
winter]. And he worked in the prison for twelve years, as it says:
"His feet were subjected to fetters... until his prediction came
true..."(Psalm 105:18-19, JPS).
[The word for "his feet", רגליו, is so spelt in the text but is to be
read as "his foot", רגלו. According to Rabbi David Luria (Pirqei
deRebi Eliezer 39:29), the midrash is deriving ten years from the
value of the extra yodh in the ketiv. For a similar drash, see the Baal haTurim, Genesis 39:20, who derives Joseph's ten years in prison - before the addition of a subsequent two - from the
extra yodh in the ketiv there.)
So, to answer your question, at the time of Joseph's arrest he was:
Seder Olam Rabbah: 18 years old;
Shemot Rabbah: 18 years old;
Pirqei deRebi Eliezer: 20 years old (acc. to R' David Luria, 18 years old).
