They couldn't have said "Let all who are hungry come and eat", because only those who were registered for the Pesach could eat at the Seder.
Did they say it (or something like it) earlier (on Erev Pesach) to invite people to join the Korban? If not, when and why was the line added to the Haggadah?
(Pesudo)-Malbim in Midrash Haggadah advances the following theory.
This passage originated in the Babylonian Exile (and thus is in Aramaic) and was recited on the afternoon before Pesach. It was meant as an invitation to paupers to join one's seder that evening, and consisted of the phrases:
הָא לַחְמָא עַנְיָא דִּי אֲכָלוּ אַבְהָתָנָא בְאַרְעָא דְמִצְרָיִם. כָּל דִכְפִין יֵיתֵי וְיֵיכֹל. הָשַּׁתָּא הָכָא, לְשָׁנָה הַבָּאָה בְּאַרְעָא דְיִשְׂרָאֵל.
This is the bread of affliction that our forefathers ate in Egypt. Whoever is hungry let him come and eat [it later]. This year we are here, next year in Israel.
After many Jews returned to Israel, the text was still recited there on erev pesach (as you suggested) and was changed to add a reference to the korban pesach and to make the final line more relevant:
הָא לַחְמָא עַנְיָא דִּי אֲכָלוּ אַבְהָתָנָא בְאַרְעָא דְמִצְרָיִם. כָּל דִכְפִין יֵיתֵי וְיֵיכֹל, כָּל דִצְרִיךְ יֵיתֵי וְיִפְסַח. הָשַּׁתָּא עַבְדֵי, לְשָׁנָה הַבָּאָה בְּנֵי חוֹרִין.
This is the bread of affliction that our forefathers ate in Egypt. Whoever is hungry let him come and eat. Whoever needs, let him join in our korban pesach. This year we are slaves, next year free people.
Our text combines both versions, and is recited at the start of the seder as a commemoration of the old practice to recite it on erev pesach.
He also presents an alternative theory, that this paragraph was always recited at the start of the seder (rather than on erev pesach) and was meant to encourage the children to come to the table for the start of the proceedings.
According to this approach, the phrase "Whoever needs, let him join in the korban pesach" is directed at the household, and is to be understood as "Whoever needs to eat from this Pesach [because he is part of the registered group] let him come and do so [now, as the seder is starting]".
