I think it's the bayis, rather than the retzuah. Here are some reasons why:
Rashi writes (Devarim 6:8):
וקשרתם לאות על ידך: אלו תפילין שבזרוע
And you shall tie them for a sign on your hand: these are the tefillin
of the forearm
It sounds like the part of the תפילין של יד that is the sign is the part that goes on the forearm; however, I'm not sure if this is a good proof, because the whole tefillin is called "tefillin of the forearm;" the word "זרוע" here might just come to differentiate this tefillin from the head-tefillin.
The בעל הטורים, there, writes:
ידך. בגי' גובה היד
Your hand: on the height (higher part) of the hand
Again, this sounds like the אות is the part on the זרוע, the forearm.
The best proof I found to say that the "sign" is the bayis is in the תרגום יונתן, chapter 6, verse 8:
וְתִקְטוּרִינוּן לְאָתִין כְּתִיבִין עַל יְדָךְ דִּשְמָאלָא
And you will tie this for a written sign on your left hand
Of the two parts you mentioned, the only one that could possibly be a "written" sign is the bayis of the tefillin.
(all translations mine)