In Chovos Halevavos Gate 3 ch.4
In what has been mentioned, it has thus been demonstrated that all
human activities belong either to those (1) commanded, (2) forbidden
or (3) [permitted which is] sufficient. For whatever is not in the
category of [permitted which is] "sufficient", and is either
superfluous or deficient, must necessarily belong to the category of
what is commanded, if done for the sake of G-d; or it belongs to the
category of the forbidden, if it is not done for His sake.
Enquiring more closely into what is "sufficient", for example, in
obtaining a livelihood, we find this too in a commandment, set forth
in the account of Creation, "And G-d blessed them, and G-d said unto
them: 'Be fruitful and multiply, fill the land and conquer it.'"
(Genesis 1:28). And it continues, "Behold, I have given you every herb
bearing seed, which is upon the face of all the earth [.. to you it
shall be for food]" (ibid. 1:29). Hence, to seek a sufficiency in food
belongs to the class of the commanded.
This being the case, it has been demonstrated that all human
activities are either in the category of what is commanded or in the
category of what is prohibited. If what is done is in the category of
the commanded, it is a good deed. If one is able to do it but neglects
his obligation, he falls short in his duty. So, too, if one does one
of the things that are prohibited, he is a sinner. If he abstains from
doing it, he is a righteous man, provided that he abstains out of fear
of G-d, as Scripture says, "They also do no iniquity. They walk in His
ways." (Tehilim 119:3).
And so, if one does one of the things permitted, in the right and
proper way, he is righteous, as the Psalmist said, "A good man is
gracious and lends; he conducts his affairs with measure" (Tehilim
112:5). If he, however, exceeds and goes beyond what is sufficient, he
falls short in his duty, because this will mislead him to what G-d
warned against. So, too, if he denies himself what is sufficient when
he is in a position to obtain it, his aim being to train himself in
the service of G-d and to reign over his lusts, so as to come nearer
to G-d or separate himself from this world and direct his attentions
to the better world hereafter, he is righteous and his conduct is
good. But if he does this not for the sake of G-d, he falls short in
the fulfillment of his duty, and his conduct is reprehensible.
Hence, human actions fall into the categories of good and bad. The
intelligent person is one who weighs his actions before he does them,
as is here set forth, examines them carefully with his mind and
recognition, chooses what is good among them and abandons what is not
good, as David, peace be unto him, said: "I thought on my ways and
turned my feet unto Your testimonies. I made haste and did not delay
to keep Your commandments" (Tehilim 119:59-60)
i.e. in context there, all human activities are either mitzva or sin. so yes, wasting time is a prohibition.
nevertheless, as he says in Gate 5 regarding not looking down at other people:
And even if his appearance is bad, it is possible that the reason is
because he is ignorant of his obligations to the Creator. Therefore he
is more pardonable than me, because my knowledge is greater than his.
For the Creator claims from a man only according to the extent of his
knowledge.
