Bamidbar Rabbah 9:6:
וְאוֹמֵר (תהלים לז, כא): לֹוֶה רָשָׁע וְלֹא יְשַׁלֵּם וגו', יֵשׁ אָדָם לֹוֶה וְאֵינוֹ פּוֹרֵעַ שֶׁאֵין הַדְּנֵיסְטֵיס מְשַׁבֵּר שִׁנָּיו וְגוֹבֶה אוֹתוֹ, מַהוּ: לֹוֶה רָשָׁע וְלֹא יְשַׁלֵּם וגו', יֵשׁ אָדָם שֶׁהוּא גוֹזֵל בַּיִת אוֹ שָׂדֶה אוֹ מָמוֹן, בֵּית דִּין מוֹצִיאִים מִמֶּנּוּ, אֲבָל אִם בָּא עַל אֵשֶׁת אִישׁ מַהוּ מְשַׁלֵּם, אִם מְשַׁלֵּם כָּךְ שֶׁגַּם הוּא יִתֵּן לוֹ אִשְׁתּוֹ לִזְנוֹת עִמָּהּ, נִמְצָא הָעוֹלָם מָלֵא מַמְזֵרִים, הֱוֵי: לֹוֶה רָשָׁע וגו', הָרָשָׁע עוֹשֶׂה מִלְוֶה שֶׁאֵינוֹ יָכוֹל לְשַׁלֵּם, שֶׁאוֹסֵר אִשָּׁה עַל בַּעְלָהּ, אֲבָל מִי שֶׁהוּא צַדִּיק אֵינוֹ לֹוֶה דָּבָר מִן חֲבֵרוֹ, אֶלָּא מַה שֶּׁהוּא חוֹנֵן וְנוֹתֵן
And it says (Tehillim 37:21), "a wicked one borrows and does not repay, [while the righteous is generous and gives]." Is there a person who borrows and does not repay, and the creditor doesn't break his teeth to collect from him?! What, then, is [the meaning of] "a wicked one borrows and does not repay"? A person may steal a house, or a field, or money, and the courts will recover it from him, but if he cohabits with a married woman, what restitution can he make [since an adulteress is forbidden to her husband]? If [the adulterer] repays in a similar manner by allowing his wife to cohabit with him, it would come out that the world would be full of mamzeirim! This is "a wicked one borrows and does not repay," for the wicked one takes a "loan" that he cannot repay, for he causes a woman to be forbidden to her husband. But one who is "righteous" only borrows from his friend something [with which the latter is] "generous and gives."
In summary, the Midrash interprets the first clause as referring to the wicked one, who takes one's wife knowing that he cannot repay, while the righteous one takes only that which is given with consent.
Given that the first clause is dealing with adultery, according to this explanation, it's reasonable to assume that the second clause is, as well. Would it not come out, then, that a righteous person may cohabit with one's wife with her husband's consent?
Yet, that is not the case. The Torah forbids eishes ish regardless of what her husband wants. So what is the correct interpretation of the Midrash?