The Chofetz Chaim in Ahavat Chessed part 2 chapter 22 writes about a borrower
He must beware not to break the terms stipulated by the lender. If he
does, he is termed a gazlan (robber). He may not use the borrowed
article for any purpose not stipulated, nor may he retain beyond the
Your case here is one of a guardian, not a borrower, but I believe the analogy holds and the guardian must return the object the first time he is being asked to (as this becomes "the alloted time").
By definition the guardian is not the owner, keeping it against the wishes of the owner is stealing. This can also be learned from the laws of lost objects (Devarim 22:2, MT Gezelah va'Avedah 11:1-2).