- אבר מן החי - They ate a limb from a live animal
- עריות - They were guilty of illicit relations
- שקורין לאחיהם עבדים - They called the sons of Bilhah and Zilpah slaves.
The Shela (Torah Shebiksav Vayeishev Tzon Yosef § 58) (see here, paragraph beginning ויבא יוסף) wants to explain how the brothers could be guilty of such sins. It was all Yosef's misunderstanding. He says that they were using the sefer yetzira, which is only for those of noble lineage. They couldn't share it with the children of Bilhah and Zilpah, as their mothers were maidservants. Yosef took this to mean they were making fun of them, but that wasn't their intention.
The brothers created an animal with sefer yetzira, which by definition doesn't need ritual slaughter. Yosef thought they were eating a limb from a regular animal, and therefore suspected them.
He also says the brothers created a women with sefer yetzira. They were walking around with her, and Yosef thought she was a real women. He therefore told their father they were engaged in illicit relations.
I don't understand this third point. Why would the fact that she's not a real person make it any more appropriate? Why would they make a woman in the first place? How is that appropriate? You're right it may not be גילוי עריות per se, but still seems very suspicious and doesn't seem to help their reputation much.
The original text:
ואפשר שבראו השבטים בצירוף אותיות מס' יצירה בראו נקיבה והיו מטיילין עמה ויוסף לא ידע מזה והיה סובר שהיא נקיבה אשה מהאב ואם ובא והודיע לאביו שהם חשודים בעריות
Note: The original Midrash says תּוֹלִין עֵינֵיהֶם בִּבְנוֹת הָאָרֶץ, they've laid their eyes on the women of the land. According to this, they weren't women of the land. I guess it depends if the issue was that they were Caananite women, or they were engaged in inappropriate relations (as Rashi seems to say).