Was Esau commanded in circumcision? For instance, if he did teshuvah, was he still obligated in circumcision for himself and his children?
The Talmud derives from a verse (B'reishis 21:12) that only some of Yitzchak's offspring would be considered Israelites (though this future nation was not given the name "Israel" yet, as Ya'akov was not yet even born). This verse excludes Esav.
The Talmud states specifically that descendents of Esav are not obligated in circumcision (Sanhedrin 59b). From a later verse (B'reishis 28:4), it appears that the Rambam also excludes Esav himself from that category (Hil. M'lachim 10:7), although the statement in that later verse happened after Esav received his own blessing (and well after he himself was presumably circumcised).1
That said, the Talmud (Kiddushin 18a) does posit that Esav may personally have had the status of a non-observant Israelite (being descended from Yitzchak as he was), and thus his inheritance of Se'ir (D'varim 2:5) doesn't prove that the laws of inheritance apply to non-Jews (so another verse is adduced to show that the laws of inheritance apply to non-Jews).
1 Pirkei d'Rabbi Eliezer (28) states that Yitzchak circumcised both Ya'akov and Esav, but Esav later spurned circumcision by not circumcising his own descendents: "ר' אומ' יצחק מל את יעקב ואת עשו, ועשו מאס ברית מילה כשם שמאס בבכורה". For an alternative view, see sabbahillel's answer.
I remember a medrash that because Eisav was so red, they thought that he would bleed too much if the circimcised him at eight days old (sakanas nefashot). As a result, they delayed until he was older. When he became older, he absolutely refused to become circumcized and his loss of the birthright cause him to be rejected. The mitzvah of circumcision became solely part of the mitzvos of Bnei Yisrael.
Rabbi Dov Kramer discusses this in his Dvar Torah on Parshas Vayishlach However, he points out that even those who say that Eisav was circumcised (whether at eight days or later) and that his descendants continued it, agree that he rejected it once Yitzchak died.
Targum Yonasan says that Yaakov’s fear was based on Eisav having the merit of honoring his father. In Iyun HaParasha #105, a question is posed based on Tosfos (on 25:25) quoting a Midrash that explains why Eisav was never circumcised. Since Eisav was very red when he was born, Yitzchok thought it would be dangerous to circumcise him on the eighth day. After a year or two, when his color remained the same, Yitzchok realized that this was Eisav’s natural complexion so he could be circumcised, but decided that since he didn’t circumcise him when he was eight days old, he would wait until he was 13, the same age Yishmael was at his circumcision (see 17:25). However, when Eisav turned 13, and Yitzchok wanted to circumcise him, Eisav refused.
It should be noted that not everyone agrees that Eisav was never circumcised, as Tana D’vay Eliyahu (24) says Esaiv’s descendants kept the mitzvah of circumcision until Yitzchok died, and Agadas B‘reishis (58:4) says that Eisav was circumcised by his parents, but subsequently rejected it (see also Pirkay d’Rebbe Eliezer 29 and B’reishis Rabbah 63:13).
Chabad.org also says this
According to the Gemara (Chullin 47b), when a child is born red, a brit cannot be performed on him until the blood in his veins settle down. Consequently, when Eisav was born, he was too red to be circumcised. When he became older and returned to a normal complexion, his father wanted to circumcise him, but Eisav refused.
(שער בת רבים - שפתי כהן, ועי' ילקוט שמעוני)
In Breishis Rabbah 76:08 it says there that HaShem said, "You refused to give your daughter (Dinah) to you circumcised brother (Aisov). Now worse will befall you! She will be taken by an uncircumcised man!"
Does this not mean that Aisov was indeed circumcised? At least later on in life? I was actually confused when I saw this as well.