Many Rishonim observe a nuanced distinction in the text. They highlight that "וּמְקַלֶּלְךָ֖" is expressed in the singular form, whereas "מְבָ֣רְכֶ֔יךָ" is written in the plural. From this grammatical distinction, they draw an interpretation that those who curse Avraham are relatively few, while those who bless him are numerous. In the Targum Yonason's interpretation, the verse is taken quite literally, suggesting that only one individual, Bilam, will be the one to curse Avraham. (see Baal Hatorim that explains: ומקללך אאור – בגימטריא זה בלעם הבא לקלל בניך.) Rabeinu Bichayeh, on the other hand, identifies this individual as Nimrod.
However, it doesn't seem like everyone liked Avraham at the time of this blessing, especially since we know that he was thrown into a furnace of fire. (see Rabeinu Bichayeh)