With the current European horse-meat scandal, those of us who strictly require all of our food to have kosher certification are looking pretty smart right now.
The food business is complicated. As Rabbi Moshe Heisler, one of the nation's foremost experts in kashrut, told me there was a time when all you had to worry about was what ingredients were used in the making of a particular food item. But now there are all sorts of agents and additives that are used in the processing of food that have nothing at all to do with its flavor and, therefore, aren't considered "ingredients." Yet, these agents and additives, e.g. food coloring, can be unkosher.
Moreover, as our communities grew, and Jews started to travel, it has become impossible to know, first hand, whether the Jewish caterer, restaurant owner, butcher, or food processor is, first of all, upright and G-d fearing. Once upon a time, you knew that much because, in a small community you knew everyone. It is impossible for us to have that kind of information today, so we must rely upon our rabbis to be our agents for this purpose. We also must rely on them to keep up on the latest kashrut information from around the globe because the food purveyor can't know everything; his day is busy enough. When my wife was a mashgicha, she was instructed to thoroughly check all of the whole chickens for any anomolies. Every two or three cases of Empire chickens, she would find entire hearts and livers still attached to the inside of the birds. Our counsel knew, from experience, that notwithstanding Empire's excellent redundencies in cleaning out the insides of birds and inspecting them afterwards, the sheer volume of chickens slaughtered daily meant that some imperfectly cleaned birds got out. Having an informed mashgiach on the premises localized the problem before the birds got cooked with hearts and livers inside.
And mistakes happen. To continue the above example, a chicken was cooked with hearts and livers still attached. The certifying rabbis and their mashgichim came through to limit the damage and also to prevent the owner from over-reacting (he might have thrown away pans that weren't really treifed.
Now in some close-knit communtiies where everyone has a pretty good knowledge of kashrus, there is more trust. I went to a small simcha in Boro Park at a Hungarian chassidic hall. I was surprised that some of the hasidic ladies had brought their own cakes to the affair. I was told that this was not abnormal there since everyone knew everyone and knew they could trust them. That wouldn't fly in my community, but it works well between certain blocks in Brooklyn.
It is unfortunate, but there are people in the business who do not fear G-d. They may try to sneak in treif. Or they may try to substitute a cheaper, uncertified oil for the one approved by the vaad. Anything to save a buck. Often these are caught by wide-awake mashgichim. In my community, the mashgiach caught the acting manager, a non-Jew, bringing in non-kosher steaks because he had run out of meat and desperately wanted to keep steaks on the menu, but had no time to get them that night from the kosher butcher. Fortunately, these kinds of incidents are rare, but when they occur, we should say "thank G-d."