I've noticed that the Friday night shemoneh esrei (nusach Ashkenaz) uses the line "בה", shacharit and musaf use "בו", and mincha uses "בם" in the phrase "וינוחו _ ישראל מקדשי שמך..."
What is the reason for this variation?
Aruch HaShulchan 268:14 explains that Shabbos is called in the Torah both masculine and feminine.
כי קדש היא לכם מחלליה מות יומת is feminine.
זכור את יום השבת לקדשו is masculine.
In the Torah there is a feminine way of referring to evening (ליל) therefore we say "בה" then. Day (יום) is only masculine in the Torah therefore we say "בו". Mincha time which is close to the evening yet is still day we say "בם" which includes both.
Matan Shabbos has additional reasons.
The Lubavitcher Rebbe (Likutey Sichos 14 pg. 18) explains based on what is explained in Kabbalah that there are three distinct levels on Shabbos: 1) The night of Shabbos 2) The day of Shabbos 3) Mincha on Shabbos afternoon. On Friday night the world receives spiritual energy from the attribute of malchus. On Shabbos day there is an even higher revelation of the middos, and on Shabbos afternoon a yet high revelation that unites malchus and middos. Thus, on Friday night we say "בה" corresponding to the feminine aspect of malchus, on Shabbos day we say "בו" as middos are considered masculine, and finally on Shabbos afternoon we say "בם" indicating the unification of malchus and middos.
According to Artscroll (quoting Rabbi (Elie?) Munk), בה refers to שבת (f.), בו refers to the יום (m.) of שבת, and בם refers to many שבתות (f. pl.), referring to the day which is fully שבת. This change is because the night represents the first seventh day, the one of creation, when it was alone without anyone keeping that day, "like a lonely woman without a husband." The morning represents it at the time the Torah was given, when it became betrothed to Israel (I don't know why it changes it from female to male, though). The minchah represents the future.