Punishment doesn't (and didn't) work the way I think you're assuming. Ancient Israel didn't have a "police force" to deal with violations of halacha. Any transgression punishable by a court was acted on only if witnesses came forward to bring an accusation. As part of doing that, the witnesses would certainly have to know if the person they accuse is Jewish. If it's not known, someone would have to ask (in court, if it wasn't resolved earlier).
So to use your example, if Shimon and Reuven both observe Ploni, who they know to be a Jew, lighting a bonfire on Shabbat, and their direct appeal to Ploni didn't work, then they would go to a beit din to testify to what they saw. Ploni and any other witnesses would also testify, and the court would make a decision. (Judges, not juries, determine guilt under this system.) There are extra stringencies for the conduct of cases that could result in the death penalty, and according to the talmud (Makkot 1:10) executions were rare.
Tractate Sanhedrin covers the conduct of courts in a fair bit of detail.