We’re commanded to be sensitive to a convert, as we know what it’s like to be in a foreign land (Shemos 23:9):
וְגֵ֖ר לֹ֣א תִלְחָ֑ץ וְאַתֶּ֗ם יְדַעְתֶּם֙ אֶת־נֶ֣פֶשׁ הַגֵּ֔ר כִּֽי־גֵרִ֥ים הֱיִיתֶ֖ם בְּאֶ֥רֶץ מִצְרָֽיִם׃
Do not oppress a convert: you know the soul of a stranger, for you were strangers in Mitzraim.
By the same logic, one would expect that we be commanded not to oppress a slave, as we know what it’s like from being slaves in Mitzraim. Yet the Rambam (Avadim 9:8) rules that, by non-Jewish slaves, one is technically allowed to oppress them:
מֻתָּר לַעֲבֹד בְּעֶבֶד כְּנַעֲנִי בְּפָרֶךְ. וְאַף עַל פִּי שֶׁהַדִּין כָּךְ מִדַּת חֲסִידוּת וְדַרְכֵי חָכְמָה שֶׁיִּהְיֶה אָדָם רַחְמָן וְרוֹדֵף צֶדֶק וְלֹא יַכְבִּיד עֵלּוֹ עַל עַבְדּוֹ וְלֹא יָצֵר לוֹ וְיַאֲכִילֵהוּ וְיַשְׁקֵהוּ מִכָּל מַאֲכָל וּמִכָּל מִשְׁתֶּה.
One is allowed to make a non-Jewish slave work with hard work. Even though this is the law, it is the the pious measure and the way of wisdom that a man be merciful, chase righteousness, not weigh his yoke on his slave, not pain him, and feed him and give him to drink from all food and drink.
The Rambam goes on to describe how the earlier Sages would give their slaves from whatever they themselves were eating, even feeding them before themselves; how they wouldn’t embarrass them, reasoning that they were sentenced to enslavement, not embarrassment; how they would listen to their sorrows and comfort them, rather than heaping pain on them; etc.
All that said: while one might be an idiot for doing so (not walking “in the way of wisdom”), the Rambam clearly writes that one is allowed to be a cruel master toward his non-Jewish slave. Why? The same way we’re supposed to be sensitive to other groups, since we know how it feels to be in that position, why isn’t there a strict requirement to be a kind master to a non-Jewish slave?