There are at least three positions most people refer to when they say mashgiagh. The following are things I've heard from my Rebbeim and fiends who are in the hashgacha business.
The person who's name or organization is on the store or taking care of an event, who, either in person or through an employee, is supposed to pop in once in a while to make sure everything is kosher v'yosher.
The person who would sit in the establishment the entire time the cooking process (or milking) is going, this is called a mashgiach temidi. Sometimes this person will be involved in the actual cooking to make it bishul yisroel according to sfardi standards.
The person who goes into a nonjewish establishment and prepares it for kosher use, such as a hotel for a wedding. This person usually then fills the role of #1 or #2 for the duration of the event, many times employing others for help.
Now, which one of these mashgichim know what?
The umbrella name of #1 its usually headed by a knowledgeable person. Many times the employees, the on-site mashgiach, know less about kashrus than the average person attending the event. They are instructed to call their supervisor if a question arises. This proves problematic when points cannot be properly conveyed over the phone, or when the event is on shabbos. Many times a shabbos morning kiddush will have boys (who I would call barely religious, but won't for fear of 'who are you to judge' comments) who you wouldn't trust to cook your eggs in your own house, two calling themselves the waiters and one calling himself the mashgiach. Next event They swap roles. This is common even by the most Chareidi/Yeshivish/ Chassidish kiddushim you could imagine.
The name on the store is also a tricky sticky point. How often does someone come inspect the store? One of the largest Jewish communities has a certain mashgiach's name on 80% of it's restaurants. This mashgiach has no employees and is left to supervise every single store himself. Reports vary, but apparently it's easier to sight Bigfoot than to see this mashgiach in one of his restaurants.
Mashgiach #2 is usually the least knowledgeable. Consider the hours they keep and the money they get paid and you'll understand we are not dealing with skilled labor. We are dealing with someone being paid to sit somewhere with the single qualification of being Jewish. (I was in a restaurant once, under the hashgacha of one of the biggest and most used kashrus organizations, which I was told was the most reliable hashgacha in Manhattan restaurants. Halfway through the meal the mashgiach temidi who obviously suffered from a mental disorder was making his way through the restaurant asking people where they live and of he could hitch a ride home. I would trust the illegal immigrant employees more, but they are not Jewish.)
On to #3. This is usually the most knowledgeable mashgiach. Fortunately or not that knowledge is often times used to invoke a leniency that other mashgichim would not rely on, which can lead to issues.
As with any other endeavor, buyer beware.
Considering Orthodox societal norms, a woman could theoretically find work as mashgiach #2, although I doubt it would happen. Perhaps if they gave the job a different title like 'eid echad/ yisroel hamivashel' it might fly.