How should one act halachically if he invites not so observant female family members (specifically mother in law) at the shabbat table and they start singing.
There's nothing you can do. Just don't pay attention to her voice (assuming, for the sake of the question, that your Rov holds kol isha is forbidden even for Shabbos zemiros). If you tell her to stop, or even hint, that would be perceived as rude, and it would alienate her from Yiddishkeit and perhaps prevent her from coming closer to observance. If she becomes more observant, she will learn on her own.
This is standard kiruv practice. Can you imagine a Chabad rabbi or rebbetzin telling the women in a Chabad house shul to stop singing? They would never do such a thing, because they know it is counterproductive. Sometimes one has to sacrifice one's own stringency for the sake of someone else's growth (The Gutnick Chumash discusses this idea.) (I understand that this is not a matter of a chumra but of different halachic opinions of differing stringency.)
In addition, correcting your mother-in-law in front of your wife is never a good idea, since your wife will likely get angry and take her side. Source for most of the above: the "Fifth Book of the Shulchan Aruch" (Common Sense).
Note that in a similar situation, when in the Israeli army troops had to attend a concert with kol isha, some rabbis (such as Lazer Brody, if I remember correctly) advised religious soldiers to simply tune it out and ignore it rather than getting up and leaving and thus causing a scene.
I would make an exception to the above advice for someone who is reasonable certain they can mention the issue (perhaps not on Shabbos or if it comes up in conversation) so that she learns about kol isha, without offending her or alienating her from Judaism. If that is the case, then he could do so.
If your rabbi holds that this type of kol isha should be avoided at all costs, regardless of other considerations, then don't invite them, or find a polite way to inform her about kol isha. An intermediate alternative would be to sing wordless niggunim or songs without pesukim, to avoid reciting pesukim in the presence of ervah.