Rav Hirsch explains on that pasuk that it was an external gathering and making a pile. That is why he (actually Rabbi Levy his grandson translating from German to English) uses the term rabble.
וְהָאסַפְסֻף in the the chumash (both Hirsch and Art Scroll as examples have no vowelization under the aleph) is similar but somewhat different from אסף as explained by Rav Hirscha about Beha'aloscha 11:4 in a way similar to Ibn Ezra.
והאספסוף – שנאספו על ישראל ואינם מהן, והן ערב רב. והמלה כפולה כמו:
סחרחר (תהלים ל״ח:י״א), חמרמרו (איכה א׳:כ׳).
As Rav Hirsch explains:
The א being quiescent gives the nuance of meaning that the absorption
was more of an external nature, that those who were "gathered in" did
not really enter into the real national being of the nation. It was
more of a ספף than an אסף. Israel was more of a סף, a container, a
vessel, and a "threshold" to them, they were "contained" in Israel but
never really became an integral part of it.
Thus, they were piled up inside the Bnai Yisrael and continued to regard themselves as separate in a way as well as insisting on getting all the benefits but not subject to any of the responsibilities.
Note that Rav Hirsch seems to say that חג האסיף would therefore come from the root אסף rather than ספף because the harvesy actually becomes part of Bnai Yisrael and requires תרומות ומעשרות to be taken and made kadosh so that the tevel of the initial in-gathering becomes permitted to the Am Hashem. Something that the rabble did not allow to happen (as shown by their actions).
I should note that it appears that not everyone of them was like this, but that those who were caused the deaths at Kivros Hata'avah. Rav Hirsch goes into more details on the matter in the entire commentary of chapter 11.