The previous answer does touch on the point that this was a declaration to excuse those placed in cherem (excommunication). See this comprehensive article that discusses the topic as well as how some of the ideas crossed over into the Kol Nidre paragraph that follows in the service. Much of the article has English translation. Feel free to contact me (comment) if you need help with specific other Hebrew words or phrases.
Excerpts of some main points:
Citing Mordechai Yoma ch. Shivat Yamim p. 247:
We enter the synagogue and lift the ban on praying with those
individuals who violated communal decrees even if those individuals
have not asked that the bans be lifted. This practice is based on what
we learned (Crisus 6b) Rabbi Shimon Chisda said: any fast day on which
sinners in the community are not asked to participate is deemed to not
be a fast day. This rule is based on the fact that the odor of
galbanum is unpleasant and yet it was included among the spices for
the incense (Kitores) used in the Beis Ha’Mikdash. Abaye said: ‘We
learn this rule from the verse (Amos 9, 6): And has founded His vault upon the Earth.
So, this paragraph answers your main question that sinners must be included in every fast day, not just Yom Kippur.
The Hebrew term used in the phrase you cited is עבריינים "ah-var-yah-nim" which is not the usual word used for sinners which would be חוטאים (though, it does translate as that). So, it seems that these were a special class of sinners.
The עבריינים are those individuals who were excommunicated from the
Jewish community because they violated rules established by the
community. Many hold the misconception that the עבריינים are the
Marranos- Jews who chose conversion over death during the Spanish
Inquisition. That the Mordechai, a student of the Maharam
M’Rottenberg, and who lived in the 1200’s, refers to the custom
provides evidence that the practice originated in Ashkenaz well before
the Spanish Inquisition.
Read the rest of the article, as it delves into more detail of how excommunication worked, then, and how this phrase as well as the Kol Nidre paragraph includes the concept of charamim and similar vows, and how this entire intro. paragraph developed.