I have personally heard from Harav Chaim Kanievsky shlit"a that the term "גזל זמן" (stealing time) [and גזל שינה - stealing sleep] is a halachic misnomer, because stealing time [or sleep] is not considered theft.
He cited the ToseftaTOSEFTA (BK 7:3) which says that there are 7 types of theft, and stealing time is not listed there. Also the Sforno in Shmos 20:13 lists kidnapping and theft of money and deceiving as included in the prohibition of "Do not steal". He does not include stealing time [or sleep]sleep*].
R' Kanievsky added that these are still forbidden under the laws of ואהבת לריעך כמוך.
*[Regarding disturbing sleep, RCK cited Baba Kama 20b which includes disturbing a neighbor's sleep in the category of נזקי שכנים - damages to neighbors].