I just thought of this answer:
The commandment to bring the Pesach the second year starts with:
וְיַעֲשׂ֧וּ בְנֵי־יִשְׂרָאֵ֛ל אֶת־הַפָּ֖סַח בְּמוֹעֲדֽוֹ׃
The Jewish people should bring the Pesach at its proper time.
Rashi says (from Sifri, which elaborates a lot more)
"במועדו" אף בשבת, "במועדו" אף בטומאה
"at its proper time" meaning even on Shabbos or in a state of tumah [if the whole nation is tamei from corpses]
These men also used the same word במועדו:
וַ֠יֹּאמְרוּ הָאֲנָשִׁ֤ים הָהֵ֙מָּה֙ אֵלָ֔יו אֲנַ֥חְנוּ טְמֵאִ֖ים לְנֶ֣פֶשׁ אָדָ֑ם לָ֣מָּה נִגָּרַ֗ע לְבִלְתִּ֨י הַקְרִ֜ב אֶת־קָרְבַּ֤ן יְהוָה֙ בְּמֹ֣עֲד֔וֹ בְּת֖וֹךְ בְּנֵ֥י יִשְׂרָאֵֽל׃
And those men said to him "We are tamei from a corpse. Why should we be left out and not be able to bring the offering to Hashem at its appointed time among the Jewish people"
It seems from their usage of this word that they were basing themselves on the permission to bring the Pesach when the entire nation is tamei, and wanted to bring it even when only their little group was tamei. Daas Zekeinim's explanation goes in this direction. (Something is messed up on Sefaria, I can't get it to display in Hebrew.)
To take it a step further: wfb mentioned in the comments that there's another drasha brought in the Gemara in Pesachim:
ופסח גופיה מנא לן אמר רבי יוחנן דאמר קרא איש איש כי יהיה טמא לנפש איש נדחה לפסח שני ואין ציבור נידחין לפסח שני אלא עבדי בטומאה
How do we know that the Pesach can be brought even in a state of tumah? Rabbi Yochanan says, because the pasuk says "If someone is tamei from a corpse..." - one person can be pushed off to Pesach Sheni, but the whole nation can't be pushed off to Pesach Sheni. Rather, they bring it even though they're tamei.
At the time when these people complained, the pasuk Rabbi Yochanan is quoting hadn't been said yet, so they may not have seen any reason to distinguish between a few people and the entire community. Without איש איש, it's perfectly reasonable to believe במועדו is a blanket permission to bring the Pesach in a state of tumah.
Since the normal allowance of tumah doesn't work for other kinds of tumah, their argument wouldn't have worked for women who were niddos.
Furthermore, it seems from the whole commandment of Pesach Sheini that it's primarily designed for corpse tumah specifically, איש איש כי יהיה טמא לנפש. Practically you can bring it for other kinds of tumah as well, but that's a secondary derivation; the main association seems to be to tumah from corpses. This may be have something to do with why women didn't complain after the fact, but could use further development.