The OP accepts that the "brothers' revenge" started with "Yossef's dreams".
However, this is not entirely the case.
The problem started with Yossef's slander against his brothers, reported to their father, Yaakov.
Gen. 37:2 "...and Joseph brought evil tales about them to their father."
The situation was then irritated further in Gen. 37:4:
"And his brothers saw that their father loved him more than all his brothers, so they hated him, and they could not speak with him peacefully."
So jealousy was another reason.
In fact, slander and jealousy continued to be the underlying reasons behind the ill feelings between Yossef and his brothers. The dreams merely added to the overall negativity of the situation.
After the first dream the verse says:
Gen. 37:8 "And they continued further to hate him on account of his dreams and on account of his words.
Rashi: "and on account of his words: Because of the evil tales that he would bring to their father."
After the second dream the verse says:
Gen. 37:11 "So his brothers envied him, but his father awaited the matter."
So we see that envy and slander were still the fuel of the hatred. The dreams were just an additional annoying catalyst.
(Contrary to the OP's assumption, we also find that Yossef was someone who could at least have "shown signs of genius" at this time:
Gen 37:3 "And Israel loved Joseph more than all his sons, because he was a son of his old age; and he made him a fine woolen coat."
Rashi; 2nd explanation: "a son of his old age: Heb. - בֶן זְקֻנִים, for he was born to him in his old age (Pirkei d’Rabbi Eliezer, ch. 38). Onkelos rendered: for he was a wise son to him. Whatever he had learned from Shem and Eber he gave over to him. ...")
So what about the verse: "וְעַתָּה לְכוּ וְנַהַרְגֵהוּ וְנַשְׁלִכֵהוּ בְּאַחַד הַבֹּרוֹת וְאָמַרְנוּ חַיָּה רָעָה אֲכָלָתְהוּ וְנִרְאֶה מַה־יִּהְיוּ חֲלֹמֹתָיו"׃
"Come now, let us kill him and throw him into one of the pits; and we can say, ‘A savage beast devoured him.’ We shall see what comes of his dreams!”
Rashi tells us that the brothers actually did not say the last part: "We shall see what comes of his dreams!"
It was said by Hashem Himself.
Rashi: "and we will see what will become of his dreams: Rabbi Isaac said, This verse says: “Expound on me.” [I.e., this verse demands a midrashic interpretation.] The Holy Spirit says thus: They (the brothers) say, “Let us kill him,” but the verse concludes: “and we will see what will become of his dreams.” Let us see whose word will stand up, yours or Mine. It is impossible that they (the brothers) are saying,“and we will see what will become of his dreams,” because, since they will kill him, his dreams will come to nought."
But even if they did mention the dreams when being annoyed at him, (in addition to the slander and jealousy) it does not prove they believed he even had a dream; or that the dreams meant anything.
Rather, it simply could mean that they were threatened by a brother who liked to advertise he was going to rule over them.