The ketubah, like any Jewish legal document, requires the signature of two witnesses. Besides being Jewish men who are unrelated to each other (or to the bride or groom), there are other qualifications.
The Shulchan Arukh (חושן משפט הלכות עדות סימן לד) rules that
רשע פסול לעדות
A rasha is invalidated from serving as a witness
What is a rasha? The Shulchan Arukh continues:
כל שעבר עבירה שחייבים עליה מלקות; ואצ"ל אם חייבים עליה מיתת ב"ד. ל"ש אם עבר לתיאבון, ל"ש אם עבר להכעיס. הגה: עבר עבירה שאין בה מלקות, פסול מדרבנן
It is anyone who transgresses a transgression that requires lashes. It's unnecessary to discuss if someone deserves death through a court sentence [i.e., since even a more minor offense disqualifies him]. It also doesn't matter if he transgressed out of appetite or anger.
Rama: If a person transgressed a transgression that doesn't require lashes, he is invalidated rabbinically.
Transgressing Shabbat observance, a major biblical commandment, would certainly fall into that category. Defining a rasha these days is complicated, but modern authorities essentially stress that a ketubah witness should be Torah-observant [committed to following all of halakha] and definitely Shabbat-observing. The Yalkut Yosef (נישואין ושידוכים פרק ז' - נתינת הטבעת ליד הכלה), for example, puts it this way:
כן יש להקפיד שהעדים יהיו יראי ה' ושומרי תורה ומצוות, ולא חלילה מחללי שבת בפרהסיא ומגלחי זקנם בתער וכדומה. ורב המסדר קידושין בחתונה אצל חילוניים, יקח עמו עד כשר ויהיו הוא עצמו והשני, שני העדים. וטוב שתלמידי חכמים יהיו עדים
One must be sure that the witnesses are God-fearing and keep the Torah and mitzvot; that they do not publicly desecrate Shabbat, shave their face with razors, and so on. The rabbi officiating a wedding for secular Jews should bring with him a kosher witness so that he and the second one will act as witnesses. It is also good for students of sages to be witnesses.
If a witness is not religiously observant, or if there is no second witness for the ketubbah signing, then the ketubbah is invalid. Whether or not a wedding conducted without a valid ketubbah can still make a couple married is a different question, and has been discussed here.
The footnote for the Yalkut Yosef provides a story related to this discussion:
והדבר מצוי בעיקר ברבנים מסדרי קידושין לחילוניים, ופעמים שלא נמצאים שם אנשים יראי שמים ושומרי תורה ומצוות זולת אלה שיש להם קירבה משפחתית לחתן או לכלה, וצריך ליזהר בזה. ומרן אאמו"ר שליט"א נוהג ברוב הפעמים שהוא עצמו משמש כאחד העדים, ומצרף עמו עד נוסף. ואחד הרבנים בארץ סיפר לנו, שפעם סידר קידושין באחד המושבים החילוניים, ולא מצא שם אפי' אדם אחד שיהיה ראוי להיות עד, ואחד הציע עצמו להיות עד, וכאשר לקחו לצד ושאלו היאך הוא מתגלח, נתברר שהוא מתגלח בתער. עד שמצא שאחד המלצרים הוא שומר תורה ומצוות, וצירף אותו יחד עמו לעדות הקידושין.
This issue is essential for rabbis who officiate at secular weddings, as sometimes there are no God-fearing, Torah and mitzvot-observant men to be found, except those related to the bride and groom [and thus disqualified]. It's important to be careful about this. Maran [R. Ovadia Yosef] often served as a witness [in addition to being the rabbi] and would bring a second witness with him. One of the rabbis in Israel told us that once he officiated in a secular community, and he couldn't find even one man that could serve as a witness. One man offered himself to be a witness, but after the rabbi took him to the side and asked him how he shaves, it became clear that he shaved with a razor [and was thus disqualified from serving as a witness]. It was only until the rabbi found a waiter who kept Torah and mitzvot, and joined as a witness with him, that he could make the wedding.