Partial answer:
The JVL's source, as already mentioned by @EraserX in the comments, is the 1901 Jewish Encyclopedia. The Jewish Encyclopedia's source is Meir (Marcus) Fischer's תולדות ישרון (Toledot Yeshurun), p. 22, who writes:
"בשמונת ימי חנוכה ששו ושמחו בישועת אלהים ויפצחו רנה לרועץ אויבי בת ציון ויהללו לשם תפארתו על אשר הוביש את צוררי יהודה עד לא יכלו קום...אמנם, לא כן היה עמהם בענין ימי הפורים, כי חוץ מערי אזילא - Azila - ועבי יאקוב - Ebi-Iacub - לא ידעו ממנם אף שמץ דבר, וכאין ואפס נחשבו להם אלה, כאשר ספרו לנו כותבי קורות הימים וסופריהם.
During the eight days of Chanukah they celebrated in great cheer and joy for God's salvation and shout aloud for the destroyer of the enemies of the Daughter of Zion, and praise His magnificent Name for His felling of the enemies of Judah until they were unable to rise...However, not so was the case regarding the days of Purim, for besides the cities of Azila and Ebi-Iacub, they did not know a single thing about them, and they were thought of as meaningless in their eyes, as we are told by the chroniclers and their scribes.
Fischer listed most of his sources in a note on pp. 12-14, however they are all in foreign languages (Arabic, Latin, Spanish, French, etc), which, even with Google Translate, makes it difficult to know from where exactly he took this particular tidbit. I skimmed a couple of the books but had no luck yet.
Two possible reasons may be hinted at in Fischer's book:
- As he stated (p. 21), in the first several centuries CE the Jews of the Mauretania region, as he calls it (a region which includes Tunisia, Morocco, and more), only followed the Written Torah and not the Oral Torah, because they did not have the Talmud. This may explain why they were only aware of some d'Rabbanan customs.1
- The statement about Chanukah and Purim is included within a list of minhagim held by the Mauretania Jews in their early years, and is immediately followed by a hatred for wine and drunkenness (pp. 22-23). It's therefore possible that they had been aware of Purim but at some point their aversion to wine and wine-based merriment led them to stop keeping Purim and eventually forgetting about it.
1 See also Rambam's letter to his son which mentions that the Jews of Tunis and other Berber lands were uneducated in the Talmud and other Chazalic sources and laws. It should be noted, however, that Rabbi Meir Mazuz, when he was younger, opined that the author of the letter (Rabbi Mazuz also voiced skepticism of its association with the Rambam) referred to the Jews between Egypt and Tunis, i.e., the Jews of Libya.