The first question: Obviously, we can present two contradicting views:
The owner is interested in such behavior because it increases the clientele (מחלקים קליות ואגוזים), or he might intentionally offer his books for such a use. Therefore he allows such a use לכתחילה and he's Mochel (forgives) the damages.
The owner is not interested and protesting to such use and considers that a Gezel but does not sue the users in court.
The resolution of such a conflict would be:
A person should always be aware of the possibility of Gezel, which is Deorayso and a very serious sin. So when in doubt, (as I understand you were in your case) one should definitely ask for explicit permission from the store owner. Since we rule that ספק דאורייתא לחומרא.
This also holds if you see others grabbing sforim freely, they might have a personal arrangement with the owner.
Where (local neighborhood/city/country) it is customary to arrange the Tefillos in book stores intentionally to draw attention or increase sales, this behavior is allowed לכתחילה.
The second question:
Theoretically, according to #1, the owner's claim might be justified, because unlike the simple case of שואל שלא מדעת (borrowing without permission) which [only] applies for items that are not for sale, for items in the store the very intention to use them shows the intention of owning.
Think about entering a pharm opening a perfume and using one once on yourself. Your intention to use is automatically translated into a purchase.