There is a concept in morality, that we do not inherently deserve1:
עַד שֶׁלֹּא נוֹצַֽרְתִּי אֵינִי כְדַאי
Before I was created, I was worthless
This is logical. We do not have the power to create ourselves, or to sustain ourselves now that we are created, nor do we have the power to create pleasures, or create the ability to enjoy pleasures. We do not inherently deserve life, not to mention pleasure.
The point is far stronger if one includes the idea of tzimtzum she'lo k'peshuto. This is the kabbalistic idea that brings with it the concept that Hashem is immanently present in the world, because the world is made, so to speak, from His Essence2.
Is it any wonder then, that Chazal who were on a very high level of sensitivity to truth and holiness, and of pristine character traits (including bashfulness), were unable to bring themselves to partake of the physical pleasures of life? They were highly in tune with the truth that we do not deserve this, similarly to how many people feel when we are guests in someone else's home, and feel we need to wait for permission to begin eating and drinking.
This is the same Chazal who also taught us about all of the mitzvot and their derivations (such as seudat mitzva). The mitzvot and halachot are the Will of Hashem. The positive side to the point presented thus far is that they would have felt that, since they don't deserve to be here, but Hashem put them here, lets them be here and wants them to be here, pleasing Him and doing whatever He wants is imperative. Again, the mashal would be being a guest in someone else's home, we feel a natural desire to wanting to please our host, and follow his rules.
So in the context of a seudat mitzva, or oneg Shabbat, Hashem Himself expects us, as part of His Will, to enjoy physical pleasures. This provides ample opportunity to be true to the point being made in Yerushalmi Kiddushin 4:12 about "In the future one will be judged for all that his eyes saw and he didn't eat".
Even if this is a general principle and not just limited to food, but all permissible worldly pleasures, Chazal would still approach this under the context of "fulfilling Hashem's Will", rather than personal satisfaction. Some may have been unable to bring themselves to do it, simply finding it too against their nature and difficult, whereas others might have been able to pull it off, and derive some minimal pleasure under duress, while avoiding doing so from a place of satisfying their nefesh or yeiter. Thus, their declaration that "I didn't partake of physical pleasure in my life" still holds true.
For the rest of us, we take the middle road. We start in a place of deriving pleasures from the world without any good middot getting in the way, and strive to reach a point where we are doing so l'halacha. Hashem did indeed create pleasure for more reasons than just giving our yeitzer hara something to do. Pleasure is a reality and has many kosher applications, many mitzva applications, can be a road to the ultimate goal of getting closer to Hashem, the source of (and only true) pleasure, and is indeed the reward in the World to Come. Getting to the point where we include Hashem in our pleasures, by recognising it comes from Him, thanking Him and praising Him, and looking after our health, our middot, and making sure that we are doing so in a way that doesn't deviate from His Will - nay is in fact an expression of His Will - is our goal and will cultivate in us the same positive middot that Chazal had, the same penimius recognition of our undeservedness, which in the right context is only a good, healthy thing!
Just like guests at a shabbat meal - if they avoided enjoying all the delicious dishes you made for them, you will hold it against them!
I've seen sources I can't recall right now that state that part of the point of having to die and be resurrected is because when we partake of physical pleasures inappropriately, we destroy a good sensitivity towards them that we will need in Olam Haba to enjoy them properly. Death and resurrection is the only way to get it back. Does anyone have a source for this as it would be pertinent to this answer. It seems very logical. Olam Haba is for those who have come to the point of "deserving" life and pleasures by earning it through Olam Hazeh. Any time we derive physical pleasure in this world, we desensitise ourselves to it. Fair enough if we are under instruction to derive that physical pleasure from Hashem, but any time we did so inappropriately, this is going to detract from the reward of Olam Haba and it is right that, after teshuva and a life of mitzvot, we are given the opportunity to redeem our sensitivity, in preparation for Olam Haba.
Note: I am not necessarily saying that the pleasures of Olam Haba include physical pleasures like food etc. But the point can still stand in a more general sense.
This idea is generally my own, therefore I'd appreciate some feedback. If people generally view it as in line with Torah thought, I would consider accepting it.
1 - Based on Rava's prayer in Berachot 17a, also brought in Yoma 87b.
2 - Rabbi Shmuel Schneersohn of Lubavitch, “Mi Komokhah” in Likutei Torah—Torat Shmuel, Sefer 5629 (Kehot Publication Society: New York, 1992) p. 163. See also Rabbi Shalom DovBer Schneersohn, Torat Shalom p. 198 “All is encompassed in G‑d’s essence, accordingly no other being exists at all. Yet, this implies that nothing was created, which is impossible to say, since creation is itself the divine name elokim. It is a divine name and therefore true. That is, elokim exists, and accordingly there is concealment (that is, there is otherly existence, which is the concealment). However the concealment is encompassed in the essence, because the divine name elokim—which is the divine power that contracts—is also of the essence…”. See also Maimonides, Misheh Torah, Laws of Foundations of the Torah 2:10: "הוּא הַיּוֹדֵעַ וְהוּא הַיָּדוּעַ וְהוּא הַדֵּעָה עַצְמָהּ הַכּל אֶחָד". See also https://www.chabad.org/library/article_cdo/aid/3432275/jewish/Do-Chabad-Teachings-Say-Anything-About-the-Mind-Body-Problem.htm