The Rambam deals with this situation as with many others Yibbum vChalitzah - Chapter Two The quoted sections below should answer your questions. It appears that the choice is completely with the brother(s). Also note that if a brother jumps in and performs chalitzah (or yibum) the other brothers cannot do anything about that. The Rambam also deals with cases in which two brothers cause problems for each other, forcing chalitzah. Chapter six deals with cases in which the brother can do either chalitzah or yibum, can do neither chalitzah or yibum, can do yibum but not chalitzah, or can do chalitzah but not yibum. The Rambam brings up a number of complicated cases.
In chapter 1 the Rambam says
If the yavam does not want to perform the rite of yibbum, or
if the woman does not consent,4 he should [free her from this
obligation through the rite of] chalitzah. [Only] afterwards is she
permitted to marry another man.
It is a positive commandment5 of Scriptural law for [a brother] to
perform chalitzah for [the deceased's widow], if he does not want to
perform the rite of yibbum, as [Deuteronomy 25:9] states: "She
shall... remove his shoe."6
The mitzvah of yibbum takes precedence over the mitzvah of chalitzah.7
This means that if the woman refuses, the man should perform chalitza, but he is able to insist on yibum even if the woman does not want to. Note that if the woman insists, halacha 10 below states that she loses her kesuvah. This is similar to the halacha of a "rebellious wife". It does appear that if the woman does not want to perform yibum and the brother is willing to let her go, that would be preferable. Note that in Chapter 2, Halacha 10,we say that we "compel" him to perform chalitzah. This means that like a get, he must still make the decision.
However, the quotes from Chapter 2 should give the basic halacha. I am not posting the footnotes, but you can see them at the site. I left the footnote number in the quotes.
When [a yavam] engages in marital relations with his yevamah,
he acquires her [as his wife]. [This applies] regardless of whether he
entered into these relations unintentionally10 or with a licentious
intent,11 under duress or willingly, whether he acted with a
licentious intent and she acted unintentionally or under duress, or
she acted with a licentious intent and he acted unintentionally or
under duress, whether she was asleep12 or awake, whether he performed
vaginal or anal intercourse,13 whether he inserted merely the head of
his penis or the entire organ.14
The laws pertaining to a yevamah who is fit to perform yibbum30 and
refuses to do so are the same as those governing a woman who rebels
against her husband.31 We compel her yavam to perform chalitzah with
her,32 and she forfeits [the money due her by virtue of her] marriage
When the deceased is survived by many wives, the one that the yavam
asks to perform yibbum and refuses is considered to have "rebelled."
He should perform chalitzah with her, and she forfeits [the money due
her by virtue of her] marriage contract.33The other wives who were not
asked [to perform yibbum] receive [the money due them by virtue of
their] marriage contracts, as other widows do.
If [the deceased was survived by] many [brothers], and the
eldest brother states that he wants to perform yibbum with [one of the
deceased's wives], [she is under obligation to him].34 Even if she
does not desire [to marry] him, but is willing to marry one of his
brothers, her desire is not considered. For the mitzvah is that the
eldest brother perform yibbum.
Halacha 12 [The following rules apply when] the eldest brother says:
"I do not want to perform either yibbum or chalitzah. Behold, my
brother is present." If one of the brothers asks her to perform
yibbum, and she does not desire [to marry] him, but she does desire
[to marry] another brother, and he desires [to marry] her, she is not
considered to have "rebelled." Once the eldest brother upon whom the
mitzvah is incumbent demurs, all the brothers are equal.35 Since she
desires [to marry] one of [the brothers], and he desires [to marry]
her, she is not considered to have "rebelled."36
Moreover, if one of the brothers was in another country, and the woman
says: "I would prefer to wait for him to come and perform yibbum with
me. I do not want [to marry] this one," she is not considered to have
"rebelled." [Since] the brother who asks [to marry her] is not the
eldest, we tell him: "If you desire to perform chalitzah and pay her
[the money due her by virtue of her] marriage contract, you may.37 If
not, she wants to wait until your brother comes. Since you do not have
a prior claim, [she is granted that prerogative].38
If the brother [whom the yevamah desired to marry] returned
and did not desire [to marry] her, we turn again to the brother who
desired to perform yibbum, but was not desired by the woman. We tell
the woman: "There is no one who desires to perform yibbum with you
except this one, and the mitzvah of yibbum is given priority. Either
perform yibbum with this one, or leave without receiving [the money
due because of your] marriage contract, as is the law with regard to
all women who rebel."