So your new reference to Prof. Amar's book picked my interest, so I took it out of the university library a couple of days ago. Turns out it's really more of a kuntress...but it has great info. I'll summarize the info that pertains to your question (I'll also note that he brings many sources from poskim and such):
It turns out that in past generations, most communities did not have the proper tools to shape cattle leather (בהמה גסה) into tefillin, and so resorted to using the leather of small flock animals (בהמה דקה - looking for a better translation of the term...). This leather could be shaped more easily, but that meant that it could also warp relatively quickly. Often the upper part of the batim would warp and become any of the following: Cuboid with rounded corners; a tube that has taken on the shape of the rolled-up scroll; a tower shape (wider on the bottom and gradually becoming narrower) - either a cuboid or a tube; or having a cuboid base and a tubular upper section. Some examples:
(picture copyright to Prof. Amar and those he lists in the kuntress)
There were also cases in which the makers were unable to - or chose not to - make the tefillin perfectly square in the first place, as seems to have been the case here:
(from the catalogue of יודאיקה ירושלים, תשמ"ו, item 116)
And of course, the Renaissance drawings of round tefillin: 1, 2, 3 (pg. 5), 4 (pg. 119).
Nonetheless, Jews continued using these tefillin, which naturally begs the question: How does that square (excuse the pun) with the gemara?
Prof. Zohar writes on pg. 17, note 52 (my translation):
"...About round tefillin, it says in the mishna: one who makes his tefillin round it will endanger him, and there is no mitzvah in this" - it seems that this is not referring to this sort of tefillin...for it is unimaginable to think that the Rishonim and Acharonim used tefillin that were unkosher. For a few suggestions, see Tefillah Le'Moshe, pg. 56-57; Habermann, pg. 176-177.
I have not checked Tefillah Le'Moshe yet, but A"M Habermann had an interesting suggestion. He suggests that originally, tefillin were only scrolls without batim, tied around the head. Later, as people wore them for longer durations during the day and there was fear that the scrolls would be ruined, batim were created1. In the time of the gemara there were sects that did not have batim for their tefillin. They would place the scrolls on their foreheads and tie them straps. He suggests that "round tefillin" meant that they would spread out the scrolls all across their foreheads in a rounded manner. Such tefillin would have been easily recognized by Roman authorities, thus endangering the wearer, while tefillin with batim would have looked more like little amulets (and apparently that's what the NT called tefillin - amulets). He believed that this makes the following statement better understood (Brachot 23a):
"Rabbi Meyasha, son of Rabbi Yehoshua ben Levi, said: The halakha in this case is that one rolls up the phylacteries in their straps like a scroll, and holds them in his hand opposite his heart."
This refers to tefillin that aren't in batim but were only scrolls with straps. Therefore, the comparison of rolling "like a scroll" makes more sense (then the commentators' suggestion that it refers only to the straps).
In short, we find that the prohibition of making round tefillin does not refer to cylindrical batim.
1 This suggestion doesn't negate the concept of "The requirement that phylacteries must be square is a halakha transmitted to Moses from Sinai" - we may say that when there are batim, then they must be square, and this was given at Sinai.