My answer is inferred from my understanding of traditional sources and discussions I have had in Yeshivah, receiving orally from my Rabbis.
This means I am offering an explanation and will provide support proofs from Tanach or from the Rabbis; but I am unaware if a single Traditional source spells out exactly what I am saying.
First we do see from Tanach that when some people realize that they have seen a "malach" (angel), they worry that they might die.
1) Judges 6:22 "Now Gideon saw that he was an angel of the L-rd, and Gideon said, "Alas, O L-rd G-d! Because I have seen the angel of the L-rd face to face."
Rashi: "Alas" is an expression of worry at seeing an angel as if to say "What will become of me?"
2) Judges 13: 21-22 "And the angel of the L-rd did not continue to appear to Manoah and to his wife. Then Manoah knew that he was an angel of the L-rd. And Manoah said to his wife, "We shall surely die, because we have seen an angel."
The Hebrew word used by Manoah for what he saw, is "Elohim", which can simply be used as G-d's name. However, Targum Yonathan translates it as "an angel of G-d"
Therefore, here too in the OP's verse in Genesis 32:31, saying Yaakov saw an "Elohim" face to face, should also be translated as an angel of G-d. (see Targum Onkelos and Targum Yonathan, who explain this here as well.)
However, the OP questions why Yaakov would be afraid of seeing an angel? Perhaps Gideon and Manoah were not so used to seeing angels? But Yaakov Avinu was probably used to seeing them often (example: The "ladder" dream, Gen. 29:12; Gen. 32:2-3) as may be inferred by the following Medrash brought by Rashi in his comment to Gen. 16:13 which speaks of Hagar's lack of fear when seeing an angel of G-d. :
"...And you should know that she (Hagar) was accustomed to seeing them (angels), because Manoah saw the angel once and said, “We will surely die,” and this one saw four, one after the other, and she was not frightened. — [from Gen. Rabbah 45:7]"
So why would Yaakov be more worried to see the face of an angel than Hagar?
First of all, lets see if there is a general danger with encountering an angel and then lets see if there is a specific issue with Yaakov and this particular angel?
A) "...in order not to afford them an opportunity to see the angels performing their destructive work as angels when there was no need for them to witness this. Watching divine beings in action and being saved by such spectacles requires Divine intervention and should therefore be avoided. Compare the awe with which Manoach comments on what he has seen in Judges 13,22, as well as Yaakov’s reaction in Genesis 32,31 after his wrestling match with the angel of Esau.
(Rashbam to Gen. 19:17 commenting on why Lot and his fleeing family were told not to look back.)
B) Exodus 23:20 "Behold, I am sending an angel before you to guard you on the way and to bring you to the place that I have prepared. Beware of him and obey him; do not rebel against him, for he will not forgive your transgression, for My Name is within him. For if you hearken to his voice and do all that I say, I will hate your enemies and oppress your adversaries. For My angel will go before you, and bring you to the Amorites, the Hittites, the Perizzites, the Canaanites, the Hivvites, and the Jebusites, and I will destroy them."
Rashi explains: "for he will not forgive your transgression: He is not accustomed to that [i.e., forgiving], for he is of the group that do not sin. And moreover, he is a messenger, and he can do only his mission. -[From Midrash Tanchuma 18]
Here we see that an angel is hard to deal with because it is only capable of pure justice; and it must simply perform its mission. Now when such a being is fully invested with "G-d's name" (Elokim is the Divine name which represents G-d's justice; and Yaakov's wrestling angel is referred to as "Elohim"), it is like facing off against justice to perfection, with no mercy and no adjournment. Now we can start to sense why someone would be afraid.
Now let's see if Yaakov Avinu had anything to be especially afraid of when confronting (face to face) this angel?
Gen 32:25 "Jacob was left alone. And a man wrestled with him until the break of dawn."
Rashi explains: "Our Rabbis of blessed memory explained that he was Esau’s guardian angel" (Genesis Rabbah 77:3).
Now, OK, so its Esau's angel, big deal right? Guess again:
"as it is said: And Jacob was left alone; and there wrestled a man with him (Genesis. 32:25). It was Samael, Esau’s guardian angel, who wanted to kill him, as is said: When he saw that he prevailed not against him, he touched the hollow of his thigh (ibid., v. 26)."
(Tanchuma Rabbah, 8:3)
OK, so the guy's name is "Samael" or the famous "Samech - Mem" (Sam). AND he wants to kill Yaakov. So, should we be impressed? How tough is this "Sam" anyway?
"And the woman beheld Samael, the angel of death, and was afraid; yet she knew that the tree was good to eat..."
(Targum Yonathan to Gen. 3:6)
So, Esau, being the wicked fellow, has somehow attracted the very primordial angel of death himself, to be his own personal bodyguard/lawyer. We also see that this frightening agent, has been allowed to target Yaakov, at night, and all alone, in the middle of nowhere, with full intent on killing him. And of course, the angel of death is probably the best guy to hire, if you want to kill someone.
But, why would Esau be able to harness G-d's justice ? what merit could he have to persuade Heaven that he should be allowed access to such an advocate? Furthermore, was Yaakov afraid of some sin?
a) Gen. 32:11 "I have become small from all the kindnesses and from all the truth that You have rendered Your servant, for with my staff I crossed this Jordan, and now I have become two camps."
Rashi: "I have become small: My merits have diminished because of the kindnesses and the truth that You have rendered me. Therefore, I fear lest I have became sullied with sin since [the time that] You promised me, and it will cause me to be delivered into Esau’s hand[s]. — [from Shab. 32a, Ta’anith 20b]
b) Gen. 27:36 "And he said, "Is it for this reason that he was named Jacob? For he has deceived me twice; he took my birthright, and behold, now he has taken my blessing." And he said, "Have you not reserved a blessing for me?"
So we see from here that Yaakov Avinu was generally afraid that his sins have taken away his own Divine protection, and that his brother Esau has always had a complaint of injustice against him for "stealing" his birthright and "stealing" their father's blessings even though Esau was the firstborn son.
In fact, the whole Genesis 32 (which culminates in the wrestling match with the angel) is the story of Yaakov facing his brother Esau after many years of absence. Yaakov knew that there were still ill feelings against him over the birthright and blessings. That is why he sends tribute to Esau as a peace offering. However, Esau responds by marching on Yaakov with military force.
The night before they meet, Esau spiritually files a claim against Yaakov in Heaven, which results in Divine justice allowing for a trial. Yaakov understands that he must "face" his brother's claim and defends himself by saying the sale of the birthright was honest and legal, plus the taking of the blessings was warranted because Esau planned to renege on his sale.
But Yaakov didn't know he would be found innocent by G-d.
The result however, was that the angel of Esau blesses Yaakov:
Gen 32:29 "And he said, "Your name shall no longer be called Jacob, but Israel, because you have striven with [an angel of] G-d and with men, and you have prevailed."
By no longer calling him Yaakov, the guardian angel of Esau admits that Yaakov did not trick Esau. (the word Yaakov can mean "trickster". see Gen. 27:36 above)
by calling Yaakov, Yisrael; he admits that Yaakov dealt honestly. (Yisrael can be read as "Yashar-El" or "The straight one of G-d".
So when Yaakov says he saw the angel's face, he means he faced his brother's spiritual and legal complaints before G-d, and he emerged innocent. As Yaakov says to Esau when they meet the next morning:
Gen 33:10 "Thereupon Jacob said, "Please no! If indeed I have found favor in your eyes, then you shall take my gift from my hand, because I have seen your face, which is like seeing the face of an angel, and you have accepted me."
R' Chama B'Rabbi Chanina said:.... He (Yaakov) said to him (Esau), "Your face looks like the face of your angel."
(Shir HaShirim Rabbah 3:6)
I hope this helps. :)