The Gemara in Megillah 3a:
מכאן סמכו של בית רבי שמבטלין תלמוד תורה ובאין לשמוע מקרא מגילה
The house of Rebbi was mevatel Torah to come and hear the Megillah
Much has been written about why the reading of the Megillah is considered bittul Torah, as the Megillah itself is seemingly a part of Torah.
The Yam Shel Shlomo explains that it is a qualitatively lower level of learning, and there is such a thing as ביטול תורה באיכות - qualitative bittul Torah. Others explain that the Mitzvah of reading the Megillah is meant to be an act of listening, not an act of learning.
Either way, even a mitzvah is called bittul Torah and requires justification (See the gemara there, which learns from a kal v'chomer that this particular Bittul Torah is acceptable). It seems that any act of not learning needs justification (I am making the assumption that the threshold of justification is more than "I was having fun;" otherwise the Gemara wouldn't very much need to tell us about reading Megillah being enough of a justification, nor to learn that it is a justification from a kal v'chomer).
The Chayei Adam Klal Aleph:
וכן בשכיבה אצ"ל שבזמן שיוכל לעסוק בתורה ובמצות לא יתגרה
בשינה לענג עצמו
And regarding lying down, it goes without saying that when one is capable of learning Torah or doing mitzvos (i.e. not too tired) he should not sleep for enjoyment
It is hard to capture the tone with one quote, but the context is that everything you do has to be for the purpose of Hashem and not because you happen to enjoy it.