Many returnees to Judaism were born out of reform or conservative
marriages. In the case of such a person, assuming their parents were
not learned in Torah whatsoever, but whose mother is Jewish and whose
marriage is recognized in civil law, would their child be considered
born out of wedlock, and therefore a mamzer?
First, we must define a mamzer - a mamzer is not someone who is born out of marriage, its someone who is born from a illicit relationship, such as a married (and not divorced) woman with another man.
In regards to civil marriages, they carry no weight in halacha - a Jewish man and woman who received a marriage certificate from the gov't does not make them halachically married. If the woman then proceeds to have a child with another man, the child is not a mamzer. A mazer would be if a Jewish man and woman got halachically married, and then the woman had a child with another man without divorcing her husband.
I do recall my 9th grade rebbi saying that there is a teshuva from R' Moshe that a Jewish man and woman living together for a long enough time with other people witnessing may constitute a marriage, with the people witnessing being the witnesses.
How would this change if the husband or wife had been previously
married?
So technically in halacha a man can have more than one wife, so the man being married to another woman would not be a problem. A woman, however, can only have one husband. So if the woman had been halachically married to a man and never got a divorce, any child from this new relationship would be a mamzer.
How about if both parents are Jewish but they signed a kesubah with
invalid witnesses?
There are 2 parts of a halachic marriage; kiddushin, where the husband gives something of value to the woman and says 'haray at mikudeshes li bitabas zu', and nisuin, where the marriage is consumated.
I would think it would still be a marriage as long as the kiddushin happened that would go through even without a kesuba, making the woman married to the man. So any child born not from that marriage would be a mamzer.
Obviously ask a rov for any real guidance.