A deleted post from a long time ago made the claim that
[W]e are [allowed to charge interest to other Jews] but not from immediate family. Even from another Jew, if he or she is not a member of our close family. Why? Because if we don't charge interest, what we have done is charity. While to our own family, it does not constitute charity but a duty.
The post itself had been posted by a former MY user with a habit of stirring controversy, to put it mildly, and it earned a negative vote score, while being severely criticized, before it was finally deleted altogether.
The literal interpretation of the verses prohibiting us from charging interest (Lev. 25:35-37 and Deut. 23:20), seems to prohibit charging interest to "your brother". This is clearly not the Halachah.
Yes, Deut. 23:21 (the very next verse) allows charging a "Nochri" but not "your brother". Is that the basis of the Limmud (and if so, where is the Limmud recorded) that the prohibition is not limited to charging interest to your immediate-family brother, but also precludes charging your co-national/co-religious "brother" (ie., your fellow Jew)?
Essentially, I'm asking how, in this instance, we know "your brother" means fellow Jew and not biological brother (if it always means fellow Jew and not biological brother, then an answer demonstrating that will suffice).