The Talmud (Menachot 35a) says:
תנא: תפילין מרובעות - הלכה למשה מסיני. אמר רב פפא: בתפרן ובאלכסונן. לימא מסייע ליה: העושה תפילתו עגולה - סכנה ואין בה מצוה; אמר רב פפא: מתני' דעבידא כי אמגוזא.
It is taught in a baraita: The requirement that phylacteries be square is a halakha transmitted to Moses from Sinai. Rav Pappa says about this halakha: Square means along their seams and their diagonals, i.e., they must be perfectly square where the compartments are sewn to the titora. The Gemara suggests: Let us say that a mishna supports this opinion (Megilla 24b): One who fashions his phylacteries in a round shape exposes himself to danger, and it does not enable him to fulfill the mitzva of phylacteries. Rav Pappa said: This is no support, as one can say that the mishna is referring to phylacteries that are fashioned like a nut, i.e., their underside is rounded, and therefore there is a danger that if he strikes his head on a wall the underside will press into his head and injure him. By contrast, if the underside is flat one might have thought that it is fit despite the fact that it is not square. Therefore, the baraita teaches that phylacteries must be square. (Sefaria)
Tosfot there already suggests that this sounds like only the stitching at the bottom needs to be perfectly square, not the leather box on top. The Orchot Chayim (Tefillin 27) and Semak (#153) quotes "some" who are strict to have the box be square as well. The Mordekhai (Halakhot Ketanot) famously (cf. Magen Avraham to OC 32:39) disapproves of "those who make the hand Tefillah in a press that is round at the top and square at the bottom".
So clearly this position was out there. It's thus not surprising to find an ancient example of it (particularly in a Geniza after the currently accepted position became dominant).
There's further debate among those who apply the "square" rule to the boxes if it suffices that they be rectangular and if the rule applies to the heights of the boxes as well. The perfect cubes you see nowadays were not the norm historically. Non-rectangular Tefillin-boxes existed until quite recently in some Eastern Jewish communities (cf. Pe'ulath Saddiq 3:205, Rav Pe'alim 4:2).