At shalosh seudos in our shul we say bircat hamazon with R'tzei because we have not yet davenned Ma'ariv. The main difference can be seen when Sunday is Rosh Chodesh. Our rabbi's psak is to say R'tzei and not Ya'ale v'yavo before Ma'ariv.
I should emphasize that this psak is because one has not yet started Ma'ariv, even though the clock time for saying ma'ariv has been reached. We are careful to actually bentsch before saying borchu and beginning ma'ariv.
Rabbi Kaganoff also deals with the question at Do I say Yaaleh Veyavo, Retzei or both? and goes into more detail on the subject.
RabbiKaganoff.com Bensching in the Dark on Rosh Chodesh deals with the question and goes into detail on the reasons.
The Rosh (Shu’t HaRosh 22:6; Pesachim 10:7) asserts that once Shabbos
is over, one cannot say Retzei.
A disputing opinion is quoted in the name of the Maharam (see Hagahos
Maimaniyos, Megillah 2:14:1), which states that a meal begun on a
holiday maintains its special mention, even when one bensches after
the day is over. Thus, when one bensches on seudah shelishis after it
is dark, one still recites Retzei. Similarly, if one’s Purim seudah
extends into the night, one still recites Al Hanissim in the
bensching. These laws apply, as well, on Yom Tov, Rosh Chodesh and
Chanukah (Shulchan Aruch, Orach Chayim 188:10). The practice, already
cited in earlier authorities, of completing the Purim seudah after the
day is over and then reciting Al Hanissim is based on this position of
the Maharam (Rema, Orach Chayim 695:3).
As we just explained, the Maharam rules that one recites Retzei on
motza’ei Shabbos for a meal that began on Shabbos. However, if someone
recited havdalah and has not yet bensched for seudah shelishis, he
must omit Retzei, since recital of havdalah ends Shabbos. The same is
true not only regarding havdalah, which clearly ends Shabbos, but even
when one does anything implying that Shabbos is over – such as
davening maariv or even simply answering Borchu, since these
activities occur only after the conclusion of Shabbos (Shu't Maharil
56). The Magen Avraham (188:17) notes that someone who davened maariv before Shabbos is over (which is halachically permitted under
extenuating circumstances) does not say Retzei when he subsequently
bensches, even though he is still required to observe Shabbos (since
it is before nightfall). This ruling is followed by the Mishnah
Berurah (188:32) and other authorities. The Magen Avraham (263:33) and
other authorities are uncertain whether one who said hamavdil bein
kodesh lechol after Shabbos is over, but has as yet not bensched after
seudah shelishis, may still say retzei.
The Rema consistently follows the position of the Maharam (Orach
Chayim 271:6; 695:3). However, it is a bit unclear how the Shulchan
The Bach (188 and 695) views the Shulchan Aruch as being inconsistent,
arguing that this last decision contradicts the position of the
Maharam, which the Shulchan Aruch himself follows in 188 and 695. The
Bach understands, as do other authorities (e.g., the Aruch Hashulchan
188:23), that, according, to the Maharam, the essential factor is when
the meal began, whereas, according to the Rosh, the determining factor
is what day it is at the moment of bensching. According to the Bach’s
understanding of the Maharam, someone who began a meal before Shabbos
and continued it into Shabbos should omit Retzei, which contradicts
the conclusion of the Shulchan Aruch. The Bach’s approach is
consistent with the ruling of the Rema.
However, other authorities contend that the Shulchan Aruch is
following the Maharam consistently, but they understand the Maharam’s
position differently from the way the Bach did. Whereas the Bach
understood the Maharam to be saying that the sole determinant is when
the meal began, they understand that either the beginning of the meal
or the time of bensching determines whether we recite the special
holiday prayer. In their opinion, if one began a meal on a holiday but
bensched only after the holiday was over, one recites the appropriate
holiday passage (Taz 188:7; Elyah Rabbah 188:20).