In order to answer such a question, we first must clarify the reason and source for shaking it off.
In theory, there could be two bases for not having the charoset as part of what one eats.
One is the gemara in Pesachim 115a about having first matza, then maror, then korech, because otherwise the rabbinic maror would nullify the Biblical matzah. By implication in Tosafot's discussion there, there is a third level of reshut, which could nullify either of the other two levels. Perhaps the charoset, if reshut, would nullify the maror, which is Rabbinic. If this is the concern, see Shlomy's answer where the Ran dismisses the concern.
However, to understand Rav Yosef Karo in Shulchan Aruch, you need to read the Tur and then Rav Yosef Karo's Bet Yosef commentary on the Tur. This forms the background from which he extracts the simple statements in Shulchan Aruch.
If we examine Tur Orach Chaim 475 (with Bet Yosef on the side), we see the Tur writes:
אח"כ יקח כזית מרור וישקענו כולו בחרוסת ולא ישהנו בתוכו שלא יתבטל טעם מרירתו ומטעם זה כתב ה"ר יונה שצריך לנער החרוסת מעליו שלא יתבטל טעמו
That the concern is about steeping maror in charoset for a long while, such that the taste would dissipate. And the Rabbenu Yona writes, for the same reason that therefore one needs to shake the charoset from upon it so that the taste should not be nullified. Nullified in this case is not the halachic nullification of earlier, of reshut nullifying derabbanan or derabbanan nullifying deorayta, but of the dissipation of the bitterness in the sweet charoset.
Bet Yosef points us to the source:
ומ"ש וישקענו כולו בחרוסת בפרק ע"פ (קטו.) אמר רב פפא ש"מ האי חסא צריך לשקועי דאי ס"ד לא צריך לשקועי נטילת ידים למה ומיהו התם בגמרא דחינן ליה דדלמא לעולם אימא לך לא צריך לשקועי וקפא מריחא מיית אלא למה לי נטילת ידים דלמא משקעו ליה וא"כ לא קי"ל כרב פפא ולכן הרי"ף והרמב"ם והרא"ש השמיטוהו אבל הרוקח הביאו ולכן כתבו רבינו:
That gemara, despite the single dot in the quote, is actually a bit later, on Pesachim 115b:
וְאָמַר רַב פָּפָּא: לָא נִישַׁהֵי אִינִישׁ מָרוֹר בַּחֲרוֹסֶת, דִּילְמָא אַגַּב חַלְיֵיהּ דְּתַבְלִין מְבַטֵּיל לֵיהּ לִמְרוֹרֵיהּ, וּבָעֵינַן טַעַם מָרוֹר, וְלֵיכָּא. אַדְבְּרֵיהּ רַב חִסְדָּא לְרַבָּנָא עוּקְבָא, וּדְרַשׁ: נָטַל יָדָיו בְּטִיבּוּל רִאשׁוֹן — נוֹטֵל יָדָיו בְּטִיבּוּל שְׁנֵי.
And Rav Pappa said: A person should not leave bitter herbs in the ḥaroset for a lengthy period of time, lest the sweetness of the spices in the ḥaroset nullify its bitterness. And the bitter herbs require a bitter taste, and they are not bitter when marinated in ḥaroset. The Gemara reports: Rav Ḥisda authorized Rabbana Ukva to deliver a lecture, and he taught: If one washed his hands for the first dipping, he should wash his hands again for the second dipping.
So when Rav Yosef Karo writes what he writes in Shulchan Aruch, he is echoing the specific chumra expressed by Rabbenu Yona.
Now, let us turn to mimetic traditions, as referenced in NJM's answer. Basically, this is a minhag. There is a trend nowadays to look in Shulchan Aruch and explicit texts, and reason out what should be done. But at the time Shulchan Aruch was introduced (and at the time Mishneh Torah was introduced), it was controversial, because it seems to centralize ruling of halacha and delegitimizing the practice of learned rabbis in each community going through the sources and seeing what appears most correct, and ruling in such way.
While Shulchan Aruch said this, endorsing the position of Rabbenu Yona that leaving the charoset on during eating would nullify the bitterness, it is quite possible that other Rishonim, or even other Acharonim, would not find this stringency persuasive or in line with Rav Pappa's idea. Just as cooking, or steeping in water (kavush kemevushal) transforms an inherent taste, so could steeping the maror in charoset for a long period. However, simply leaving the charoset on top of the maror would not cause the intrinsic bitterness to leave. You would taste the bitterness of the maror, and you would taste the sweetness of the charoset.
It could be that in certain communities, following an understanding of halacha that was not in the particular text we are examining, they held otherwise. (And, for the reason described above.) And, despite the arising of the Shulchan Aruch to prominence, they kept their halachic practices as a mimetic tradition.