Interesting question! But Moshe need not have literally been judging all day long for Yisro's point to be legitimate.
נָבֹל תִּבֹּל גַּם אַתָּה גַּם הָעָם הַזֶּה אֲשֶׁר עִמָּךְ כִּי כָבֵד מִמְּךָ הַדָּבָר לֹא תוּכַל עֲשֹׂהוּ לְבַדֶּךָ
In the order of the conversation, the point made by Moshe immediately before Yisro's admonition is "when people have an issue I arbitrate between them and inform them of God's laws and statutes". Although the morning-till-night demand was the introduction to this conversation, it isn't explicit in Rashi's comment that it is what Yisro was responding to. The problem he saw may have simply been the large number of people bearing down.
If "נָבֹל תִּבֹּל" doesn't mean "you will wear down", but "you will get confused" (cf. Chizkuni) - i.e. between people/cases due to the volume he is processing - then the absolute number of people could be more of a contributing factor than hours spent per day. Then Yisro's advice fits right where it appears in context. But Rashi does not believe that is the meaning of "נ.ב.ל" (at least not here).
It is interesting to note that the prooftexts Rashi gives to corroborate his opinion that the operative root means "worn down" are both withered leaf metaphors. That is, a kind of decay that comes not from overuse but from disuse. Add to that the fact that Yisro is concerned not only for Moshe but also for the populace - "גַּם אַתָּה גַּם הָעָם הַזֶּה" - and it seems like Yisro is suggesting that the time it would take Moshe to deal with them all alone would be inordinate, leading them all to wither away in the queue. If so, then the logical suggestion of splitting up the work may not have made Moshe work any less hard, but improved the throughput of the operation nonetheless.
It is also worth noting the premise of Rashi's question, which is a paraphrase of Rav Chiya bar Rav Midifti's question in the g'mara:
?וכי תעלה על דעתך שמשה יושב ודן כל היום כולו? תורתו מתי נעשית
Could one possibly think that Moshe was sitting in judgment the whole day? [Then] when did he learn Torah?
As well as its resolution - quoted in the question - that the honest work of a judge is so fundamental to the functioning of the world that high quality execution thereof is not "all in a day's work", but "all in a day's work". So there is no reason to believe that Moshe would have been working any less hard or to any less effect in his capacity as judge before and after this reorganization since a) the question assumes his non judging time was occupied as well and b) as far as that d'rasha is concerned, he was presumably just as "מִן הַבֹּקֶר עַד הָעָרֶב" after as before.