North American shuls are generally partnerships created by their members, and they hire their Rabbonim. If they specify a clause that all halachik/minhag decisions are up to the Rav, then that would give him the prerogative. Otherwise, it would seem to be up to the members.
However, minhagim frequently have the status of vows which are un-nullifiable. Based on the sources below, there seem to be strict limitations to this flexibility of decision:
- Matters of minhag are obligatory even when there is no clear reason why one option is a stringency over the other, eg. switching nusach.
- If the members of the shul had a common place of origin or other reason to have specific minhagim, they would have the halacha of their original place and all newcomers would be obliged to publicly conform (anecdotally it seems obvious that such groups are allowed to start a shul in conformance to their minhagim even in a place that broadly does otherwise).
- In cases of unknowns, there is no reason to be stringent unless the stringency is known to be common practice.
Some apparent practical outcomes:
- Changing any aspect of nusach tefila is at best extremely difficult to allow. I would have thought that if the shul didn't sing Anaim Zmirot, that could be considered a stringency as much as saying it and therefore adding a song such as Yedid Nefesh to which there is no known halachik objection would be fine - but not Yigdal for example. R'Moshe explicitly says this is not the case, and adding new things which were not previously said is fine as long as there was no known explicit reason why the community did not say it.
- If there's no known reason why Anaim Zmirot was sung before shacharit, it could be moved to after, since that's the common practice. But not the other way (maybe unless there's a strong basis that that was also widespread).
- If the previous Rav or the community on its own set times for tefila and the new Rav wants to be more stringent, it would seem he can make the change as long as it's not against widespread custom. Eg. starting maariv after Shabbat at R'Moshe's tzeit rather than 35 minutes. But not in the direction of being more lenient.
- If the previous Rav stated the reason for a particular custom or decision, it should not be changed (to leniency), since it becomes a vow.
- General decisions about kashrut etc. can be changed
- In a stringent direction if there is good reason and it's not known that the community had a valid original minhag to be lenient
- In a lenient direction if the leniency is generally accepted, the previous Rav did not publicize a reason for being strict, and it's not known that the community had an original custom to be strict.
- Based on the laws of vows, if the community explicitly stated it was not accepting any previous stringent minhagim as a vow, they would not be binding.
Background: minhagim have the status of unalterable vows; one must keep his own minhagim, but may not practice either stringency or leniency in a place with a different minhag to prevent arguments. One classic source for these laws is OC 468.4: "One who goes from place which does [work on Erev Pesach] to a place which doesn't should not do [work] in a [Jewish] settlement because of arguments, but he may do in the wilderness; one who goes from a place which does not do to a place which does may not do - we give him the stringencies of the place he left and of the place to which he went. Even so, he should not be haughty before them that he is idle for a person must not act differently from others lest he cause argument". The Mishna Brura (on 468.3, s.k. 10) there notes that "Even their children after them are obligated in their stringencies till the end of all generations because, 'Do not forsake your mother's Torah'; even nullification of vows does not help in this issue".
Several modifications/expansions/clarifications of this basic principal from R'Moshe:
- Rav Moshe (OC 4.34) strongly objects to allowing newcomers to publicly practice stringencies or any visible change in local minhagim - apparently he is referring to a newly hired Rav (or similar) because he mentions removing the newcomer from his post. Even changes between equal options (eg. between nusachot) must be treated as having stringencies, even if we don't know what they are.
- In contrast, R'Moshe allows a newly appointed slaughterhouse Rav to change the process of checking animals (YD 1.13) even if the original Rav considered his approach a stringency; R'Moshe explicitly says the community of customers are not bound to the original Rav, and we see his instructions as relevant only to the specific animals he handled, i.e. the current situation is an entirely new case, as long as the original Rav did not publicly state he was requiring the particular method as a stringency or safeguard (this seems to be based on Magen Avraham 468 s.k. 4).
- R'Moshe also limits (ibid.) the concept of "the place" (אתריה) of the previous Rav to world-class Poskim (i.e. an obligation to stringency based on geographic proximity).
- R'Moshe considers (YD 2.15)the general halacha for America to be whatever was common in Europe. If a stringency was generally kept, it would be obligatory. But without clear proof this was the case (here, for inflation-testing every animal's lungs) we will not impose it.
- R'Moshe does not allow changing a minhag based on a majority (OC 2.21) as long as there remain original members of the previous minhag [citing Chok Yaakov (OC 468 s.k. 9) - I did not find this explicitly there]; it seems the intention is anyone on whom the previous minhag has effect, as the contrast is when the previous community no longer exists - if the shul is continuously used, every new person is presumably bound to the old minhag.
- In addition, R'Moshe (ibid.) does not consider a "negative" minhag to be binding even though it may imply a stringency. So not saying Hallel on Pesach night, or not saying Shir HaMaalos on Yom Kippur after Yishtabach are all things that can be changed.
I have seen nothing specifically on this question and don't think there are many poskim who will discuss it except for 20th-century Americans, since shul structure in Europe was so different (more likely for a town to have an established and unquestionable Rav, with several shuls operating for various parts of the community) and is so different in Israel (many minyanim without any Rav, but the official "Rav HaIr" is almost guaranteed not to be anyone's personal Rav). I did not see anything more specific than what I have mentioned in the first seven volumes of Igrot Moshe (don't have 8, and have been instructed by R'Reuven to ignore it).
I would be happy to hear feedback and see suggestions for further sources.