Obviously the fact that the food is kosher does not mean that you have to eat it :) However, the halachah tells us how to determine whether a certain food item is kosher: if an otherwise observant Jew tells us that it is kosher, it is determined to be so.
There are two principal exceptions to chezkas kashrus:
1) If you have personal knowledge that the Jew who testified is not observant or otherwise not reliable, then you may not rely on his testimony that the food is kosher. This does not extend to believing unsubstantiated rumors about someone else, which probably run into the category of lashon hara in the first place.
2) If the person who is testifying is incapable of properly determining the veracity of his testimony. For example, if a person who has no knowledge of how to properly check vegetables for infestation tells you that he has cleaned the vegetable, there is no reason - or permission - to believe his "testimony". The same would apply to someone who follows different standards of kashrus; for example, someone who keeps yoshon might not eat food with an otherwise reliable hechsher if that hechsher does not determine whether products are yoshon.