I assume you were looking at the Celestial Alphabet. Note that Agrippa explicitly stated that he found these forms of writings among Jews:
"Amongst the Hebrews I find more fashions of Characters...there is also writing which they call Malachim or Melachim..." (Of Occult Philosophy, III, p. 439)
Where does this style of writing come from? This style is known as "Charaktēres" in Greek and is a mystical form of writing that has appeared in several cultures (including non-Jewish sources) since at least the first centuries CE. Letters of this style have also appeared in Jewish works, such as Sefer Ha'razim (Heb. ed. Margolioth, p. 86):

Sefer Ha'razim refers to them by the Hebraicized Greek term "כרקטירים" or "כלקטירים".
Such a form of writing may perhaps also be found in a letter by one of the Gaonim from the Genizah (MS. T-S Ar. 18 (1v)):

The Gaon wrote that this form of writing was that of the Torah when it was first given to Yisrael, but is now used only by the Kutim (Samaritans).1
For more information on this style of writing, see D. Frankfurter, 'The Magic of Writing in Mediterranean Antiquity', pp. 648-656. On Jewish sources that mention or feature this style of writing, such as the Geonic letter, see Frankfurter, ibid; Y. Weinshtok, אלפא ביתא של מטטרון ופירושה, Tmirin (טמירין) vol. 2, pp. 51-61 (available on Otzar Hachochmah).
The particular form of script known as "the Alphabet of Metatron" is at least a couple of centuries older than Agrippa's script. It is thought to originate among Chassidei Ashkenaz. Moshe Idel believes that the text that features and interprets this alphabet was written by Rabbi Nechemiah ben Shlomo Ha'navi, who lived around the time of Rabbi Eliezer of Worms. In fact, he mentions Agrippa's work explicitly (my translation):
"The influence of the Alphabet of Angels can be well-observed by one of the greatest magicians of the 16th century, Heinrich Cornelius Agrippa of Nettesheim who collected in his work De Occulta Philosophia several magical alphabets, which he attributed to Kabbalists. The second one, which he referred to as "the Alpha-Beta of Angels", has many similarities with Rabbi Nechemiah's alphabet, and the first one has similarities with 'The Alpha-Beta of Metatron'..." (הפירושים לאלפא-ביתא של מטטרון מאת ר' נחמיה בן שלמה הנביא: עיונים נוספים, p. 546).
In short, this particular form of writing is from a Jewish source. However, the exact origins of the style in general appear to still be unknown.
1 Yitzchak Ben Tzvi wrote that this Gaon was Rav Hai Gaon and that he copied the Samaritan Cursive Script of his era. That explains why it doesn't look like the Samaritan Script as it appears in Samaritan texts today.