Hanging signs saying the above was not enough to make people in my synagogue put back the s'farim they have used into the bookcases.
What other mitzvos could I bring on my next edition of the signs?
Kraina D'Igrasa 1:59 - Rabbi Yaakov Yisrael Kanievsky Zatzal writes that one who does not return the Seforim to its place is doing a Midah Achzorios and is a Rasha.
Alternatively you can print one of the pictures in this link and show what the Shul can look like if Sefarim are not returned to their place, or even better just do this a few times until the message seeps in.
I was once in a Beis Medrash which used the following tactic. They had signs up which said that the sefarim were only available to use on condition that you return them to the shelf. So if you don't return them, you have borrowed them without permission and are a thief. So you could add לא תגזול to your next sign if you would like to go this route.
Your question seems to indicate that people aren't returning their sefarim to their places, rather than taking them from the בית מדרש. If, on the other hand, you have a problem with people taking home the ספרים, you can, for a price, implement an "anti-theft" system which uses RFID tags inside of the book-covers and a scanner system by the exits which would set off an alarm (similar to a department store) every time one of the shul's ספרים exits.
For your actual problem of people not returning ספרים after learning, you could put up a fairly inexpensive camera system and record during the week. Choose a random day, go through the footage, find your usual suspects, and confront them privately about their habits.
If this doesn't work, simply post signs that indicate that there will be a penalty of $5 per ספר not returned to its place. Give a week grace period, then go through the footage for the following week and send out the bills. Receiving a penalty by mail means that you're serious.
I usually find that "guilty encouragement" is a good tactic among Jews, esp. in shuls.
An example in this case, you could either put up a sign or better have verbal announcements by you, the rabbi or whoever has authority emphasizing these ideas:
The sefarim are meant to help everyone who wants to increase their Torah knowledge. Studying Torah is listed as the one Mitzvah that is equivalent to all the other mitzvoth. We encourage you to borrow only the Sefarim that you need and return them to their place when you are finished. When Sefarim are not returned, it creates a disorderly Bet Midrash which makes the environment uncomfortable for you and your fellow learners and it prevents them from locating the materials that they need for their studies. Please avoid creating a Hilul Hashem by helping to keep this Bet Midrash neat and orderly and encouraging communal Harbatzat Hatorah.
The above is a a sample. Edit as you need. Hatzlacha.
Originally, my browser didn't display your sign. Now, that I see it, I have a slightly different take ...
$18 seems too little if you're trying to replace one of those giant Masechtot of Gemara, esp. since it's somewhat challenging replacing part of something usually sold as a set. I would raise the price to at least $54! If people actually pay up, in a sense, your shul is in "decent" shape!
Problem is, as I infer, no one wants to either fess up that they were the culprit, and even if you did catch someone (care to spend more money to place video surveillance? Maybe you need to do that!) try getting them to pay up!
Only other "safe" option is to turn your Bet Midrash into a "library". It means that when someone comes into the Bet Midrash, he has to "check in" and leave you a deposit - cash or even better - valuable ID such as a driver's license. When the person leaves, you or someone else must follow him and verify that he returns all his Sefarim to the proper places before he gets his ID back.
I have seen some large Batei Midrashim do something like this. The Sefarim are in a locked case, and a desk librarian asks for ID before anyone can borrow anything. It may involve hiring some staff, but, the cost may be worthwhile.