1. Where does the principle of halachic marriage annulment (afka'inhu) apply?
There are five Talmudic cases where this principle is invoked:
A man betrothed (eirusin) a minor (rabinically) and was attempting to finalize the marriage (nissuin) once she reached majority. Before he did so, a second man seized her, and seemingly betrothed her on a biblical level. The second man's actions are invalid, because of afka'inhu (according to Rav Ashi). (Yevamot 110a)
If a man coerces a woman to accept his betrothal, we invoke the principle of afka'inhu (according to Mar bar Rav Ashi) and the betrothal does not take effect. (Bava Batra 48b)
A man divorces his wife on condition that he does not return, and he is prevented from returning unavoidably (b'oneis). Biblically, the divorce does not take effect, but the Rabbis applied afka'inhu to retroactively annul the marriage. (Ketubot 3a)
A man sends an agent to divorce his wife, and then cancels the agency (under certain circumstances). Biblically, if the agent were to present his wife with the get she is not divorced, but the Rabbis applied afka'inhu to retroactively annul the marriage. (Gittin 33a)
A man at death's door divorces his wife and then recovers. Biblically, she is not divorced, but (according to Rabbah and Rava) the Rabbis applied afka'inhu to retroactively annul the marriage. (Gittin 73a)
Can we apply it in other cases?
Rashba in Responsum 1185 writes that we cannot:
אין לך בכלל דברים אלו אלא מה שהתירו בהן בפירוש שאם כן אף אנו נאמר שכל שעשה שלא כהוגן בקידושיו לא ניחוש לקידושין
You only have in all these matters what they explicitly permitted. Otherwise, we could also say that in every case where one acts improperly in betrothal we can ignore it.
Chacham Tzvi makes the same point in Responsum 124, although he does note that Rivash in Responsum 399 thought that it may be possible for a community nowadays to legislate that any marriages performed against its own specific rules would be invalid.
(Parenthetically, the Rabbinical Assembly's Beit Din (Conservative Judaism) will annul marriages for reasons other than the Talmudic ones, normally in agunah cases where all other methods of resolution have failed - see e.g. here.)
2. Does afka'inhu make them liable for the prohibitions involved in engaging in sex outside of marriage?
This question is a little difficult to answer. There will certainly be no earthly consequences. Even Rambam who believes that there is a biblical prohibition couldn't prescribe any punishment here, as they were not 'warned' (hatra'ah) before the act.
Is G-d going to punish them? Presumably not, as they are now following Rabbinic legislation as they were commanded to, but (not being privy to G-d's point-scoring system) I don't say this with any great confidence.
3. What is the status of the children of an annulled marriage?
The children have the status of any other child born from unmarried parents i.e. they are completely normal Jews. They are not mamzerim, who are only formed from the product of adulterous or incestuous unions (Rambam, Hilchot Issurei Bi'ah 15:1). If he is a kohein, she does not become a zonah (Rambam, Hilchot Issurei Bi'ah 18:2), and thus the children are not challalim (Rambam, Hilchot Issurei Bi'ah 19:1).
4. Can afka'inhu revoke the status of a mamzer?
In theory, it appears that applying afka'inhu would allow a mamzer to lose his stigma.
However, R. Tam held that if it is obvious that someone is doing something in order to activate afka'inhu and thus 'purify' a mamzer, then the Rabbis would not apply afka'inhu in such a case. If, however, it's clear to us that such is not his intent, then applying afka'inhu would indeed 'purify' the mamzer. (Tosafot Gittin 33a s.v. ve'afka'inhu)