It is well known the all Jacob's sons had incredible mental, spiritual and physical strength. Just to mention a few: Shimon and Levi killed all the men in Schem, Judah and the rest of the brothers could destroy all Egypt. In addition, it is said that Joseph can extinct Esav by his own (והיה בית יעקב אש ובית יוסף להבה ובית עשו לקש ודלקו בהם ואכלום ולא יהיה שריד Ovadia 1:18). So why Jacob was so afraid from a fight against Esav and his 400 people. He could easily destroy them with his sons?
The Midrash provides an answer.
In Bereish Rabbah 76:2, the Midrash notes that he was frightened of two facts; that in the time he had been away from his parent's home Eisav could have amassed a great deal of merit from both living for an extended period in the land of Israel as well as continuously honouring their father which would allow him to be victorious.
The Midrash writes:
אָמַר אִם הוּא מִתְגַּבֵּר עָלַי, הוֹרְגֵנִי, וְאִם אֲנִי מִתְגַּבֵּר עָלָיו אֲנִי הוֹרְגוֹ, הֲדָא הוּא וַיִּירָא שֶׁלֹא יַהֲרֹג וַיֵּצֶר לוֹ שֶׁלֹא יֵהָרֵג. אָמַר כָּל הַשָּׁנִים הַלָּלוּ יוֹשֵׁב בְּאֶרֶץ יִשְׂרָאֵל, תֹּאמַר שֶׁהוּא בָּא עָלַי מִכֹּחַ יְשִׁיבַת אֶרֶץ יִשְׂרָאֵל. כָּל הַשָּׁנִים הַלָּלוּ הוּא יוֹשֵׁב וּמְכַבֵּד אֶת הוֹרָיו, תֹּאמַר שֶׁהוּא בָּא עָלַי מִכֹּחַ כִּבּוּד אָב וָאֵם, שֶׁהֲרֵי כָּךְ אָמַר (בראשית כז, מא): יִקְרְבוּ יְמֵי אֵבֶל אָבִי, תֹּאמַר שֶׁמֵּת אוֹתוֹ זָקֵן וּבָא עָלַי לְהָרְגֵנִי
He said, if he overpowers me he will kill me, and if I overpower him I will kill him. He was fearful as he did not want to be killed and he was distressed as he did not want to kill. He said, all these years, he has been living in the land of Israel, and so he will come to me from the strength of living in the land of Israel. All these years he has sat and honoured his parents, maybe he will come (also) emboldened with the strength of honouring his father and mother - as it says: "The days of my father's death has drawn near, perhaps the elder (Yitzchak) has died and he is coming to kill me."