Is there a real halachic source for the widespread practice not to speak while wearing tefillin? Of course, one is required to keep his mind on the tefillin, and may not get drawn into conversations that will distract him. But what about short, non-frivolous statements? Very often these fall into the category of gemillus chasadim; e.g., wishing someone good morning, giving directions, informing what time mincha is, etc. I wonder whether there is a basis to forgo these chesed opportunities on account of what seems to be merely a safeguard rather than halachah.
וכל הזהיר במצוות תפילין, לנהוג בהם קדושה, שלא לדבר בהם דברי הבלים ושיחת חולין, מאריך ימים ומובטח שהוא בן עולם הבא, שנאמר: "ה' עליהם" (שנושאין עליהם שם ה' בתפילין) – יחיו, ולכל בהן חיי רוחי, ותחלימני והחייני"
And anyone who is careful with the Commandment of tefilin, to behave with them in holiness, not to speak, while wearing them, inconsequential words or non-sacred discussions, increases his days and is promised to be a member of the World to Come, as it says: "[If] God is upon them [that they bear the name of God upon themselves in tefilin - Kitzur], they shall live, and for all of them the life of my spirit, and You cured me and gave me life." (Isaiah 38:16)
(Free translation, with help on the Isaiah verse from chabad.org, except on the first few words, which the Kitzur is interpreting homiletically and differently from their plain translation.)
When I became Bar Mitzva I put on Tefilin by the Bobover Rebbe (Reb Shlomo ZATZAL). He was known never to speak in his Tefilin. I was so nervous that at the end of Davening1 when I took off my Tefilin I dropped it. He came over to me while still wearing his Tefilin and told me "Zorgst Aich Nisht, Es Iz Ah Gitte Siman Az Di Farstheist Dem Kedisha Fin Dem Tefilin." ("Don't worry, It is a good sign that you appreciate the holiness of the Tefilin.") When I told over the story to some people they could not get over that the Rebbe spoke in his Tefilin as he never did. However that is the Gadlus2 of this story: even though he never spoke in his Tefilin, when he saw a young Bar Mitzva boy that was close to being destroyed by the fact his Tefilin fell down on the day of his Bar Mitzva, he went ahead and gave up his lifelong Chumra3 of not speaking in his Tefilin to make me feel good.
1. Prayer service
3. Stringent practice