The torah actually does permit taking a poor man's garment as pledge for a loan. However, Devarim 24:13 says it must be returned to him at night -- he needs it to sleep in. The text here is not specific about the type of garment, but from context it sounds like it would be a wrap, cloak, or other item that could serve as a blanket.
וְאִם-אִישׁ עָנִי, הוּא--לֹא תִשְׁכַּב, בַּעֲבֹטוֹ.
הָשֵׁב תָּשִׁיב לוֹ אֶת-הַעֲבוֹט כְּבוֹא הַשֶּׁמֶשׁ, וְשָׁכַב בְּשַׂלְמָתוֹ וּבֵרְכֶךָּ; וּלְךָ תִּהְיֶה צְדָקָה, לִפְנֵי יְהוָה אֱלֹהֶיךָ.
12 And if he be a poor man, thou shalt not sleep with his pledge;
13 thou shalt surely restore to him the pledge when the sun goeth down, that he may sleep in his garment, and bless thee; and it shall be righteousness unto thee before the LORD thy God.
It's only when we get to "garment" in 13 that the pledge (that's been discussed for a few verses) is characterized. Someone with more resources would pledge something else; a poor man has only the garment off his back.
I'm not aware of any connection to the tallit beyond that they are both garments. Rabbinically tzitzit, the fringes on a tallit, are only worn during the day time, so a garment used for sleeping is probably not a tallit.