I remember hearing of a source that states that Yosef was sold into prostitution, or some other form of sexual slavery, by his brothers, and not just plain slavery as is commonly assumed. Does such a source exist?
The Jewish Virtual Library on Homosexuality states that
the Talmud records that the Egyptian Potiphar purchased Joseph "for himself " (Sotah 13b), that is, for homosexual purposes (Rashi).
ArtScroll Gemara Sotah 13b2 and 14a1
Potiphar, a courtier of Pharoah, purchased him Rav said: This means that [Potiphar] bought [Yoseph] for himself, i.e. for his own illicit purposes (as Rashi explains). However, Gavriel came and emasculated him. This is derived as follows: Initially, his name is written as "Potiphar", but in the end it is writtenn as Potiphera The name change alludes to his transformation into a eunuch.
Since the brothers did not sell him in Egypt (and as explained elsewhere, they did not even pull him out of the pit to sell him), they cannot be blamed for the purpose Potiphar originally had in mind when he bought him.
Not that they sold him into the sex trade, but that they thought that is where he ended up due to his beauty. See Bereishis Rabba 91 6 here, when the brothers came to Mitzraim for food, they were also keeping their eyes out to find Yosef. They headed to the area where the prostitutes were, saying our brother Yosef is is beautiful, perhaps he is there. And as Matnas Kehuna explains, perhaps he was sold there to his embarrassment.
(This precludes thinking he was there of his own accord. The term used for the prostitutes area is 'Kuba', which usually means a place for prostitutes who are held there against their will.)
Perhaps what you are referring to may emanate from Daat Zkenim on Genesis 39:19? (excerpt):
כדברים האלה וגו.
פירש"י בשעת תשמיש אמר' לו כן. ותימה להרב משה דבמס' סוטה פריך כתיב פוטיפר וכתיב פוטיפרע בתחלה נקרא פוטיפר וכשקנה יוסף קנהו למשכב זכור ונסתרס ונקרא פוטיפרע אלמא לא היה משמש מטתו וי"ל בשעת תשמיש לאו דוקא אלא בשעה שעוסק בעניני תשמיש. ועוד י"ל שלא נסתרס לגמרי אך כשהיה רוצה להזדקק עם יוסף היה מסתרס
כדברים האלה, “things like this;” according to Rashi, Potiphar’s wife showed him how Joseph had tried to arouse her while she was in bed with her husband. This appears difficult, as Rabbi Moshe, quoting the Talmud in tractate Sotah, folio 13 pointed out that the name Potiphar is spelled in the Torah once as פוטיפר, (39,1) without the letter ע at the end, and on another occasion with that letter ע missing (37,36) The Talmud concludes that once he had become a eunuch, as a close servant of Pharaoh, he could not indulge in sexual activity, so that Rashi’s commentary seems forced, to say the least. and additionally there are those that say that he was not a eunuch completely but when he wanted to have relations with Yosef he became impotent.
According to this, it doesn't state that the brothers sold him for prostitution, but rather that Potiphar, the last person who bought him, bought him for sexual purposes.