At what point in history did most religous people stop wearing Tefilin all day?
It's not clear. Of course even today there are a few rare people who do so.
The Shulchan Aruch Orech Chaim 37:2 brings that it no longer the custom because of the difficulties of maintaining the proper focus and self-control all day, so by then (mid 1550's) it was clearly not the typical practice.
In Halacha 25 of the Rambam's Mishneh Torah Tefillin, Mezuzah, v'Sefer Torah Chapter 4 ..."a person should try to wear [tefillin] throughout the entire day, for this is the mitzvah associated with them."
The implication of "a person should try" is that this was no longer a universal custom but still something one should do if possible. He also brings down that Rav was praised for always wearing Tefillin. We don't tend to praise people for doing something that everybody does.
The notes on the page I linked to mention the following:
"the Hagahot Maimoniot relates in the name of Rav Amram Gaon:
We saw the Geonim, the heads of the court, and the giants of the previous generations... who would not remove their tefillin until after... the Shema of the evening service." Which implies that during the time of the Gaonim only the leading Torah scholars followed this practice.
I would suppose that the practice of wearing Tefillin all day slowly died out over time until sometime during the era of the Rishonim when even Gedolim no longer followed it.
Note: even though the practice of wearing Tefillin all day is no longer commonly practices, it has not gone away altogether. In one of the local yeshivot where I live, the Rosh Yeshiva and a significant percentage of the men there where tefillin all day long (in the bet midrash as well as when going to the store or picking up their kids from school).