Prophecy and all, man is still flesh-and-blood. To know the mind of G-d would to be G-d.
What of all the "deep spiritual sixth sense" and the like? Well I guess a really evil faker can fool that too. That's how there's balance in the world. (R' Menachem Mendel of Kotzk observes that Ethics of the Fathers tells you what differentiates students of Father Abraham from students of the villainous Bilaam. "Why talk about the students and not them?", asked the Kotzer. "Because the average person wouldn't be able to tell the difference at first glance between Abraham and Bilaam")
There are plenty of stories regarding the Chafetz Chaim (Rabbi Israel Meir Kagan, lived in Russia about a hundred years ago) which indicate that he was "tapped in" to a greater source of knowledge -- just the senses he had, or things he'd referred to. Fine. But as we're humans, it doesn't work 100% of the time. It can't. When a brilliant charlatan "discovered" some long-lost volumes of the Talmud (pdf), many rabbis were skeptical -- but the Chafetz Chaim was fooled.
The Gemara in the last chapter of Pesachim talks about certain things the way the world works; despite fancy technology, these are all still fundamentals of human civilization. One of them is "that a person can't truly know what's in another person's heart."
Now clearly Esav was a more rough-cut person than his brother, there was no doubt about that. His father figured that one brother would engage the world, and assist the other brother who would be engrossed in spirituality. It was a reasonable idea, but he didn't realize how evil Esav was -- or that Yaakov had the ability to outgrow his shell and engage the world too.