B"H
I am actually writing a sefer on this (and more), so I've got this ready :)
The Rabbeinu Bahya (sv. לכו לסבלותיכם) explains:
The Egyptians, i.e. Pharaoh, respected Joseph’s law that religious
dignitaries were not owned by Pharaoh. Pharaoh himself was interested
in the people having religious leadership and instructions that would
make them a disciplined people.
Tiferes Yonasan, brought down by Rabbi Avraham Bukspan in Classics & Beyond (vol. 2) (see also answer above from Shmuel Goldstein) explains that Pharao excepted Shevet Levi from slavery because of the principle נושא בעול עם חבירו and he wanted to prevent that in combination with the coming of redemption.
Similary, the Chizkuni writes:
!לכו לסבלותיכם, “go back to your labours! “The reason why the tribe of
Levi had been exempted from performing slave labour when the Egyptians
first began to enslave the Israelites was that Pharaoh pretended to
share that burden with them in order to encourage them to perform
these labours for patriotic reasons, building fortifications against
potential invaders.
At that time the tribe of Levi remembered what their founding father Yaakov had said to them on his death bed, that Levi was not to be one of his pall bearers as that tribe was destined in the future to carry the Holy Ark on its shoulders. They therefore declined at the time to participate in the building of fortifications, and Pharaoh did not make an issue of this as also the Egyptians had a caste of priests who Joseph during the years of famine had completely excused from any taxation.
See also Daas Zkenim (ad loc.) and the Tur HaAroch (ad loc.):
לכו לסבלותיכם, “go about your own business!” Rashi equates סבלותיכם with עבודתכם, “your work,” as opposed to סבלות, forced labour, as the members of the tribe of Levi were exempt from forced labour. Ibn Ezra does not perceive the word סבלותיכם as addressed to Moses and Aaron, but as referring to the forced labour of the masses of the Israelites whom they claimed to represent. Some commentators claim that even the ordinary Israelites did not have to join the workforce until they had reached the age of nine, and that they could “retire” at the age of 60. Nachmanides writes that according to the plain meaning of the words Pharaoh told Moses and Aaron to rather perform their duties for their King, together with the remainder of the nation, seeing that on that occasion Moses and Aaron appeared before him accompanied by the common people. Pharaoh therefore told those people to get back to work instead of wasting their time dreaming of an illusory redemption. When Moses and Aaron, instead of complying with Pharaoh’s command, appeared before him once more, he commanded them to legitimize themselves by means of a miracle. Once Moses and Aaron had done so, he no longer looked upon them as part of the common people but respected them as elders, as sages. Once they had begun to decree plagues upon Egypt, he no longer treated them only with deference, but with fear, with dread. It appears that not all of the Israelites were slaving away making and laying bricks all the time, for if so the country would have become a large heap of bricks. Making bricks was a tax imposed by Pharaoh on the entire population, but the Israelites were singled out by working far longer hours and by having to supply a much larger percentage of their manpower.