It's obvious that there's a long tradition of putting on tefillin each morning that isn't Shabbos or YomTov but what's the source for this? Why each day? Before pekudei d'zmrah, after putting on tefillin, we read two (consecutive) passages from Shemos which identify tefillin as a commemoration of the yetziyah and imply that wearing tefillin is something we should do at Pesach. It says (Shemos 13) that at Pesach we eat matzah for seven days and then it talks about laying tefillin and then it says this is an annual statute from year to year. It looks like it's saying that tefillin is an annual ritual like eating matzah.
The makor for the daily (constant actually) obligation of putting on Tefillin was answered thoroughly here:
Why does one have to put on Tefillin every day?
Now, to address specifically you’re point about Pesach Rabbi Akiva in Menachos (36b) answers you’re question:
דתניא (שמות יג, י) ושמרת את החוקה הזאת למועדה מימים ימימה ימים ולא לילות מימים ולא כל ימים פרט לשבתות וימים טובים דברי רבי יוסי הגלילי ר"ע אומר לא נאמר חוקה זו אלא לפסח בלבד As it is taught in a baraita with regard to the end of the passage of the Torah that discusses both the mitzvot of the Paschal offering and phylacteries: “And you shall observe this ordinance in its season from year [miyamim] to year” (Exodus 13:10). This indicates that these mitzvot apply during the days [yamim] but not during the nights. Furthermore, the letter mem, meaning from, in the term: “From year [miyamim],” teaches: These mitzvot apply on some days, but not on all days. This excludes Shabbatot and Festivals, on which phylacteries are not worn. This is the statement of Rabbi Yosei HaGelili. Rabbi Akiva says: This verse, mentioning an ordinance, is stated only with regard to the Paschal offering, and it is not referring to phylacteries at all. Evidently, Rabbi Yosei HaGelili, who says that at night one is exempt from the obligation of donning phylacteries, says that on Shabbat one is exempt as well.
נפקא ליה מהיכא דנפקא ליה לר' עקיבא דתניא ר' עקיבא אומר יכול יניח אדם תפילין בשבתות ובימים טובים ת"ל (שמות יג, טז) והיה לאות על ידך ולטוטפת בין עיניך מי שצריכין אות יצאו שבתות וימים טובים שהן גופן אות The Gemara answers: Rabba bar Rav Huna derives the exemption from the obligation to don phylacteries on Shabbat from a different source, the source where Rabbi Akiva derives it from, as it is taught in a baraita that Rabbi Akiva says: One might have thought that a person should don phylacteries on Shabbatot and Festivals. To counter this, the verse states: “And it shall be for a sign for you on your arm, and for a remembrance between your eyes, so that God’s law shall be in your mouth; for with a strong arm God brought you out of Egypt” (Exodus 13:9). This teaches that the obligation to don phylacteries applies when the Jewish people require a sign to assert their status as God’s nation, i.e., during the week. This serves to exclude Shabbatot and Festivals, as they themselves are signs of the Jewish people’s status as God’s nation and a remembrance of the exodus from Egypt. Consequently, no further sign is required on these days. https://www.sefaria.org/Menachot.36b.5
Rabbi Akiva says “it doesn’t say chuka zos except for regarding Pesach”. ר''ע אומר לא נאמרה חוקה זאת אלא לענין פסח בלבד.