The Gemara (Yevamot 14a) says (regarding if Beis Shammai followed their own Psak):
ומ"ד עשו קרינן כאן (דברים יד, א) לא תתגודדו לא תעשו אגודות אגודות אמר אביי כי אמרינן לא תתגודדו כגון שתי בתי דינים בעיר אחת הללו מורים כדברי ב"ש והללו מורים כדברי ב"ה אבל שתי בתי דינים בשתי עיירות לית לן בה
And yet the question remains: According to the one who said that Beit Shammai acted in accordance with their opinion, we should read here: “You shall not cut yourselves” (Deuteronomy 14:1), which is interpreted to mean: Do not become numerous factions. Abaye said: When we say that the prohibition: “You shall not cut yourselves” applies, we are referring to a case where two courts are located in one city, and these rule in accordance with the statement of Beit Shammai and those rule in accordance with the statement of Beit Hillel. However, with regard to two courts located in two different cities, we have no problem with it.
אמר ליה רבא והא ב"ש וב"ה כשתי בתי דינים בעיר אחת דמי אלא אמר רבא כי אמרינן לא תתגודדו כגון ב"ד בעיר אחת פלג מורין כדברי ב"ש ופלג מורין כדברי
ב"ה אבל שתי בתי דינין בעיר אחת לית לן בה
Rava said to him: But the dispute between Beit Shammai and Beit Hillel is considered like a case of two courts in one city, as these two schools of thought were found everywhere, not in any specific place. Rather, Rava said: When we say that the prohibition: “You shall not cut yourselves” applies, we are referring to a case where there is a court in one city, a section of which rules in accordance with the statement of Beit Shammai and another section rules in accordance with the statement of Beit Hillel. However, with regard to two courts located in one city, we have no problem with it.
But, as traditionally, Sefardim were located in different cities than Ashkenazim, there's no issue.
The problem is what happens when they are in one city. So you'd say "Machlokes Abaye VeRava? Halacha KeRava except for Yal KeGam (which this isn't one of them)".
The problem is that the Rambam rules like Abaye here:
ובכלל אזהרה זה שלא יהיו שני בתי דינין בעיר אחת זה נוהג כמנהג זה וזה נוהג כמנהג אחר. שדבר זה גורם למחלוקות גדולות שנאמר לא תתגודדו לא תעשו אגודות אגודות:
So the famous question is why?
So there are a few answers:
- The Kesef Mishna answers that "Halacha follows Rava" is only true when they argue themselves. When they argue about others' opinion, there are no rules.
- The Kesef Mishna quotes R' Dovid Cohen who interprets the Rambam differently, and it follows Rava.
- The Radvaz says that Abaye only talks about small or unknown courts, where following different opinions would cause argument. There's no problem if the Machlokes is well known (Like Beis Hillel or Beis Shammai).
All except for answer #1 allow one to follow their Minhag even in a different community.