I once heard an explanation - will have to see if I can find the source - that these other servants (Jews, presumably) held with the opinion later expressed by Rambam (Hil. Yesodei Hatorah 5:1,4) that one who risks his life to keep mitzvos when not required to do so (i.e., when it's not one of the "big three," the non-Jew is doing it for his own benefit, it's not in public, and not at a time of religious persecution) is "liable for his own death."
In this case, then, the bowing wasn't being commanded as an act of idolatrous homage (or, even if it was, it might have been a case of עובד מיראה - see Tosafos to Sanhedrin 61b, ד"ה רבא). They thus protested that Mordechai is transgressing the halachah in endangering his life for this purpose.
(His reply, then, was אשר הוא יהודי - he is a prominent Jew and as such has to hold himself to a higher standard to make a public kiddush Hashem - see the second answer of Tosafos ibid.)
ואמרו חכמינו ז"ל במדרש כל מקום שנאמר המלך סתם הוא הקדוש ברוך הוא וכל מקום שנאמר אחשורוש באחשורוש ממש מדבר
. No idea what midrash he's referring to, though. Maybe he's just generalizing the above references from Esther Rabba.