The Kol Bo (21) brings the custom to request permission before zimmun:
ומדרך המוסר לכל מי שיברך שיטול רשות מן הגדולים המסובים שם
This custom to say 'ברשות' before zimmun is found when someone who has the first rights to zimmun is present, like a baal habayis, father or a kohein.
Because women are not obligated in zimmun, there would seemingly be no reason to request their permission.
R' Yitzchok Zilberstein (Chashukei Chemed, Berachos pg 276) writes regarding three sons sitting by their mother, that the mevarech should say birshus imi morasi. Although a woman is not obligated in zimmun and therefore has no precedence, there is still a different reason to request permission from the mother. This is because the Rambam writes:
ולא ישב לפני רבו עד שיאמר לו שב, ולא יעמוד מלפניו עד שיאמר עמוד, או עד
שיטול רשות לעמוד
Because the laws of kibbud em are similiar to kibbud rabo (see Pischei Teshuva YD 240:6), this would apply equally to one sitting with his mother. Therefore, there is reason to request permission even though she is not obligated in zimmun.