In the halachma anya of the Passover Hagada it says "Let all who are hungry come and eat".
Why don't we say this before the kidush?
Makes more sense that we should invite people before starting the kidush.
It's not an invitation to your Seder. It's a reminder\quote from the times of the Beit haMikdash (Temple), inviting people to share with you the Korban Pesach (Paschal Sacrifice), and the accompanying Matzo and Marror. That's why Ha Lachma Anya concludes with a prayer that we should return to Israel, and be able to once again bring the Korban Pesach.
'HA LACHMA ANYA' The opening paragraph of MAGGID - 'ha lachma anya..' is definitely not the story of Yetziat Mitzrayim, but rather a quick explanation to the guests about the MATZA on the table. Let's explain why: In the opening sentence, the leader of the Seder explains how this 'special bread' on the table is what our forefathers ate in Egypt; then he quotes what our forefathers said to one another in Egypt as they prepared to partake in the first Korban Pesach. 'kol dichfin' - reflects how they invited one another to join a common group to eat the korban Pesach (see Shmot 12:3-6); 'hashta hacha' reflects their expression of hope that by next year they would no longer be slaves in Egypt, but rather a free people living in the land of Israel. As we will explain later on, this quote of what our forefathers said to one another in preparation for the very first 'seder' in Jewish History is thematically very important, for at the end of Maggid, we will express our need to feel as though 'we were there' ('bchor dor v'dor')!
Rabbi Yehonasan Eybeschutz explains that "Kol Dichfin yesay veyechal..." is not to be interpreted as an invitation, but rather as a statement of fact about the Seder. It should not be translated (as you did) "let all who are hungry...," but rather, "all who are hungry...". It is simply adding another detail about the Seder, namely, that we invite poor people, not that this is the invitation. In fact, if one does interpret it as an invitation, than there is a far more obvious question than why we do it after kiddush: Why do we do it where nobody will hear, inside our homes? Why do we not announce it tithe street or something?
The main purpose of the Seder is to teach people and ourselves about Pesach and all the miracles Hashem preforms for us in every generation. The main time for this is during Magid. Therefor before we start Magid which is "The Seder" with ha lachma Anya and we try to include as many people as possible. Something like Porisumey Nisah. The reason why we dont go out to the market place to announce it is because unlike Chanukah where we want to spread Hashem's glory to world Pesach our focus to build our Emunah within ourselves and our own family.