Rabbi Samson Raphael Hirsch, in responsa Shemesh Marpeh, was asked regarding an influx of Eastern-European Jews to Frankfurt whose men had the custom to use the mikvah. He ruled that their custom was a wonderful thing, but if it caused one woman to not use the mikvah, it wasn't worth it.
This was cited by Rabbi Moshe Feinstein regarding what appears to have been a massive uproar in Detroit several decades ago regarding a men's mikvah. Rabbi Feinstein ruled that all it takes to ban men from the mikvah is for one woman to say: "it makes it hard for me." (We pray every day, "G-d, please don't put me to the test.")
He ruled that it would be a good thing to have a separate men's mikvah; even those who don't use it should support it as it's a good communal feature. The next question was whether to situate the men's mikvah someplace far away, or if it could just be built on the opposite side of the women's mikvah (let's say separate entrances and parking lots, but still lower costs). There he simply said that if the same location doesn't pose difficulty for the women, do that as it's cheaper. If it still poses problems, then put it elsewhere.
From Alex's comment -
R' Hirsch's opinion is mentioned in the Artscroll biography of him, and sourced to Shemesh Marpeh p. 23.
[R' Moshe Feinstein's letter is here:] Igros Moshe: Yoreh De'ah, part 2, no. 90.