If a Jewish woman marries a non-Jewish man and has children with him, are there any restrictions related to halacha or minhag on her [Orthodox] siblings' and their families' interactions with her, her husband, and her children? For example, is there any restriction on her siblings' children and her children playing together?
I think that my answer to Invited to Engagement of Girl intending to marry out: Attend/Not-attend? also applies here. The quote is from A People of Destiny
Note that "The Rav" referred to below is Joseph B. Soloveitchik who is called "The Rav" by those who learned at Yeshiva Rabbi Yitzchak Elchanan (Yeshiva University).
Of course as Rabbi Avigdor Miller, Rabbi Berel Wein, and others point out, a rav who is an expert in situations of this type should be consulted about the specific case. However, since the man in question is not and cannot be considered to be married to her (no matter what she thinks) one should be very careful to not let it be thought that the situation is being accepted.
I would think that this is more of a hashkafic (world-view) issue than an actual halachic (legal) issue. The main issue, as I see it, is about influence - how the children will interact, and who will influence who, and how much. It is also highly dependent on the people in question. For example, it is very different if the intermarried family intends to raise their children as Jews as opposed to Christian or secular.
A similar situation can be found in our tradition: Rashi (Genesis 32:23) quotes a Midrash that says that Yaakov should have married his daughter Dina to Esau, in the hopes that Dina would be a good influence on Esav and cause him to repent. However, Yaakov did not do so, perhaps fearing that Esau would be a bad influence on Dina, and cause her to sin.
The lesson is that one needs to carefully evaluate the situation, and use their judgement to determine whether the interactions will be beneficial (hopefully for everyone).
There is also the issue that suddenly cutting off all contact with one's sister and her family is likely to cause tension and strife, which will probably be worse than occasional interactions. I'd recommend that the families get together every so often, perhaps on the holidays, which provide a good framework and potentially beneficial topic of conversation. But again, this needs to be evaluated on a case-by-case basis.
As this case is deals with intimate family issues, I highly recommend that a person is such a situation speak to a rabbi who knows the families well and can properly evaluate the relevant halachic literature in light of their particular circumstances. It is unlikely that there is a clear-cut halachic ruling on this issue, and I'm very hesitant to start quoting rulings and responsa that will probably not address each family's exact circumstances.