Let's say you are part of a minyan during a holiday like Sukkot. There is only one Kohen present, so he goes up to perform the birchas kohanim. However, you know - either from reliable sources or even from seeing it yourself - that this kohen is either ignorant (i.e. he doesn't know how to daven properly, and can barely say the proper words of the blessing) and/or is not orthodox (e.g. he came to the shul in his car). What is the procedure here? Should you allow yourself to be blessed by this kohen? Should you walk out just before birchas kohanim?
Shulchan Aruch O.C. Siman 128 discuses the qualifications of a Cohen to recite bircas kohanim. Disqualification include having consumed too much alcohol, having a severe speech impediment, blindness, having taken a human life, having married a disqualifying wife (such as a divorcee) and the recent death of a close relation. The Shulchan Aruch (Sif 39) writes that in absence of aforementioned disqualifying factors, any Kohen may (and is in fact obligated) to recite the blessing, even if he is not scrupulous in his Torah observance as other sins do not invalidate him. The Mishna Berura (S"K 143) writes that this extends even to serious sins such as sexual misconduct. The Shulchan Aruch Harav (S"K 52) brings in parentheses that an exception to this is one who publicly desecrates Shabbos who has the status of a gentile and is thus disqualified from recited bircas kohanim.
The Rambam addresses your concern directly, in Hilkhot Birkat Cohanim, chapter 15, law 7:
ואל תתמה ותאמר, ומה תועיל ברכת הדיוט זה--שאין קיבול הברכה תלוי בכוהנים, אלא בהקדוש ברוך הוא: שנאמר 'ושמו את שמי, על בני ישראל; ואני, אברכם' --הכוהנים עושים מצוה שנצטוו בה, והקדוש ברוך הוא ברחמיו מברך את ישראל כחפצו
"Do not be perplexed and say, 'What effectiveness can the blessing of this ordinary/not-scholarly/not exemplary person have?' For the receiving of the blessing is not contingent on the Cohanim but on the Holy-One-Blessed-is-He. As it says 'And they shall put my name upon the children of Israel; and I will bless them.' The Cohanim do the commandment they were given and the Holy-One-Blessed-is-He, in his compassion, blesses Israel, according to his desire."
I.e., the section of the Torah that explains the commandment of Birkat Cohanim concludes (Numbers 6:27): "ושמו את שמי על בני ישראל ואני אברכם" "And they shall put my name upon the children of Israel; and I will bless them." God is the "I" who is speaking; it is always God who blesses. Birkat Cohanim is a mitzvah like any other: the purpose of performing a mitzvah is to perform a mitzvah.