This seems to start with an idea brought by the Chok Ya'akov, although he doesn't quite describe the current practice which you mention.
He writes (in 490:2) that the prevalent custom (in his day and location) was like that mentioned by the Rema in Orach Chaim 25:13, i.e. to remove the tefillin on Chol Hamoed before the amidah of Musaph.
The Chok Ya'akov explains that the reason for this is due to the impropriety of reading the Torah reading for the third day of Pesach (which contains the command to wear tefillin twice) without wearing tefillin:
איך יסיר התפילין קודם קריאת הפרשה הזאת
How can one remove his tefillin before reading this portion?
He goes on to say the since one keeps tefillin on until after the Torah reading on the third day of Pesach, one keeps them on on every day of Chol Hamoed until after the Torah reading, due to the principle of lo plug (avoiding making distinctions).
It seems like the custom you describe is to take into account this concern of the Chok Ya'akov with regard to the Torah reading on the third day of Pesach, but not to extend it via lo plug to all other days of Chol Hamoed.