Although from the gemara it seems like a regular drasha from a pasuk, meaning don't blame this on the Talmud, there seems to be an interesting explenation that I'd like to point out.
The way the עץ יוסף on the Medrash Tanchuma explains it, this is actually a device for self preservation.
In parshas Mishpatim on the words אם כסף תלוה, the Tanchuma (#15) spends much time explaining how people should give and lend to each other. The medrash then points out how these acts must be done without embarrassing the poor person, שהוא עמי, darshaning the word ׳my nation׳ as if it has two חיריק and is pronounced ׳with me׳.
The Tanchuma says this is the source of the drasha ענייך ועניי עירך, ענייך קודם. עניי עירך ועניי עיר אחרת, עניי עירך קודמין לעניי עיר אחרת.
The medrash goes on to say that Hashem is in charge and if the rich man won't be kind to the poor man, Hashem will reverse their roles. The Medrash brings a pasuk calling them brothers, proving their equality, ending with the words לפיכך, את העני עמך.
Upon these words the Eitz Yosef says the medrash is trying to convey the point that the rich man should be kind to the poor man so that the poor man should in turn be kind to him, for being poor is a turning wheel in the world and today or tomorrow he or his children will need help.
I think it's obvious from the flow of the medrash and the intent that this insurance policy of kindness will work better the closer the investments stay to home. Which is why we find varying levels of whom to give, family members, fellow citizens of your city, etc.