This text was written when the Temple stood and people were eating sacrifices. In fact, all sacrificial bread (other than a few from the Thanksgiving offering) were non-leavened, year-round. Furthermore, on any other run-of-the-mill night of the year you'd see Jews in Jerusalem eating a Thanksgiving sacrifice, and along with it some leavened bread, and some non-leavened bread. (Lev. 7:12).
The Gemara in Pesachim does discuss someone staring at a loaf and saying "hm is this leavened?", implying that unleavened bread was generally more common then; but I think the historical-context answer is the simplest.
(Similarly, the original text of Ma Nishtana had: "on other nights we eat the meat however, but tonight only roasted" -- again, ordinary sacrificial meat can be fried, roasted, boiled, whatever; the Passover sacrifice is unique in its instructions how ordinary Jews consume it. This stanza was modified when we stopped having the sacrifice; however, stanza #1, which addresses matza, was left in-place as it was the only tangible biblical mitzva still in force.)