(Based on Shulchan Aruch OC 196 with Mishna Berura)
One who ate non kosher food when not permitted to do so (ie not in a case of danger to life) should not say a bracha before or after eating and does not combine to form a zimmun. The Mishna Berura adds that some achronim hold that if it was eaten beshogeg (accidentally) then one could say a after bracha but not combine to a zimmun. This assumes the food is forbidden even rabbinically. In your case this would depend on how you categorize the actions of non religious Jews.
He notes a further case where the different people at the meal will not partake of the other persons food because of a chumra (such as one who keeps pas yisrael with one who does not) or because of different statuses (such as a yisrael eating with a kohein who is eating his terumah), in this case as long as everyone can partake of at least one person's food, they do combine.
So if you wouldn't eat their rolls as a chumra and they would eat yours, then you do combine, but if you wouldn't eat their rolls midina (by law) then you would then have to hold that they don't combine with you, although they may have to bentch by themselves.
Finally I note that if you think they are obligated to bentch (depending, as above) then even if there was no combining for zimmun you might want to bentch out loud so that they can be yotzei by hearing you. The Shulchan Aruch rules (OC 193:1) that even in a case without zimmun (ie only two people) then one should still motzi the other if the other is not knowledgeable in the text of bentching. (The Mishna Berura there notes that many hold this is true even if the listener doesn't understand Hebrew.) Obviously, this should only be done if it will not cause strife and dislike between you and the others.