Shach al ha-Torah (to Ex. 16:17) states that the figure of one omer per person was actually averaged out per family. In other words, he says, adults might end up with more than an omer apiece (since one omer wouldn't be enough for them), while the children would get less - but in aggregate, there were as many omerim as members of the family.
Ibn Ezra (to Ex. 16:15) says that "it is reasonable to say the figure of one omer per person is only for adults, while children would receive according to what they could eat."
Chasam Sofer (Orach Chaim 181) also argues that indeed the size was proportional to the person, so that an infant would get a smaller omer than an adult, and indeed that the size of it would grow along with him from day to day.
On the other hand, Noda Biyehudah (Kamma, Orach Chaim 38) states that indeed it was the same for everyone. Presumably, according to him the babies and others who couldn't eat a full omer would have had some left over. (They weren't allowed to leave any over with the intention of eating it the next day - Ex. 16:19 - but that doesn't mean that there couldn't be some that didn't get eaten and had to be discarded.)
The Lubavitcher Rebbe (Likkutei Sichos 26:103ff) explains that the man in fact had two different qualities in it. One one hand, it related to the essence of the soul, which is the same for all Jews, and therefore from that point of view it was the same amount (an omer) for everyone. On the other hand, the omer is "one-tenth of an ephah" (Ex. 16:36), implying that it's not an absolute measurement but a relative one (just as the omer varies with the size of the ephah), and in that sense it therefore varied with the abilities of the person eating it. (He notes, also, that this is why the Torah leaves that definition of the omer until the end of the passage, after describing how they ate the man at the borders of Eretz Yisrael: because it's only then that this second, individualized aspect of it became the primary one - when they were preparing for their individualized service of Hashem each in their own homestead.)