The Tur, Even Haezer 169 (Hilchos Chalitzah), says that even where Yibum is not allowed, the parts of Chalitzah like "He refuses to do yibum" and "I do not want to marry her" must be said:
ואפי' אם היא מאותן שחולצות ולא מתייבמות שלא התירו להתייבם אפ"ה צריכה קריאה
The Beis Yosef there brings this from Teshuvas HaRosh 52:2 (who says so even though the procedure would work without it, "אע"ג דקריאה לא מעכבא") and Teshuvas HaRamban. They explain how to fit the situation into the words they say, and the Ramban adds:
וגזירת הכתוב שכל חולצת תקרא מאן יבמי ושיקרא הוא לא חפצתי לקחתה
I assume the same is true for the parts at the end, which seem to criticize the yavam, "Thus shall be done to the man who refuses to build his brother's house..."
I vaguely recall that there was some Rishon who questioned this, but I can't find it. Does someone know such a source?
Addendum: The language of the Teshuvas HaRosh there leans heavily on the fact that he holds that all these cases are really עולה ליבום מה"ת. I'm wondering if even he would say it would be different according to the hava amina on daf chaf, where either because of a posuk יבמתו or because אין עשה דוחה ל"ת ועשה, there are cases where there is no yibum from the Torah, just chalitzah. And some poskim hold that way even in the maskanah, or for some other cases like sotah and machzir gerushas, see the Rashba there from the BeHag. Once the Torah divides into two cases, perhaps it would be easier to say that those parts of the procedure are for the stam case where they could have done yibum?
