Your question boils down to two parts:
- Is there a problem with opening the door at that time?
I have not stumbled upon any source that says that your door must be locked and closed for the entirety of the Seder.
- Why don't we open the door at Ha Lachma Anya when we invite people in, saying "Kol Dichfin Yeisei Veyeichol"?
This (excellent IMHO) question is addressed by Rav Yaakov Shapira (here), and he quotes Rav Kook in Olas Reiyah, that we obviously need to invite guests earlier on than Ha Lachma Anya, but we are just emphasizing a major point that we gained from Yetzias Mitzrayim, which was the Middah of Chessed:
הרב זצ'ל (עולת ראיה ח'ב עמ' רסב-רסד) מסביר שאין הכוונה באמירה להזמין אורח בפועל, את זה באמת צריך לעשות קודם לכן. אלא הכוונה להבליט את הנקודה המרכזית שקבלנו ביציאת מצרים, את מידת החסד.
Based on this, the reason why we don't open the door at this time is because we don't expect anyone to be coming at this point, we are just making this declaration about our earlier feelings, but not actually making an announcement to everyone to come in at this time. Therefore, there is no significant reason to open the door at this time any more than any other time.
Why we open for Shefoch Chamascha is (in my opinion) another, totally different question, which if I do not find on MY, I will ask.