On what basis did Judah have the right to sentence Tamar to be burned (Genesis 38)? What capital offense did she commit?
Her trial was done by the fact that she was obviously pregnant and it was known that she had not married the son of Yehuda. Thus, as the daughter of a priest who had committed znus she was subject to being burned.
and let her be burned: Ephraim Miksha’ah said in the name of Rabbi Meir: She was the daughter of Shem, who was a priest. Therefore, they sentenced her to be burned. [From Gen. Rabbah 85:10]
Ramban says that she was judged guilty of מורדת במלכות and מבזה את אביה הכוהן which allowed Yedudah (as the מלך) to decree her punishment
אך כמבזה את המלכות ועל כן כתוב ויאמר יהודה הוציאוה ותשרף כי באו לפניו לעשות בה ככל אשר יצוה והוא חייב אותה מיתה למעלת המלכות ושפט אותה כמחללת את אביה לכבוד כהונתו
None. Baal HaTurim says "burning her" meant branding, not execution.
If we accept Baal HaTurim's premise that the practice pre-Sinai was to punish failure-to-keep-the-family-name-alive with branding, i.e. permanent humiliation, you can see how the Torah instituted a far less extreme form of that with chalitzah -- a public shaming, with no corporeal punishment.