In Halichos Shlomo 7:4, Rav Shlomo Zalman Auerbach zt"l notes:
מה שנתפשט שמסבבין לאסוף מעות לצדקה גם בזמן שאסור להפסיק, לא טוב הדבר, כי ע"י כך מבטלים את המתפללים מתפלתם ומכוונתם
Those that spread out and go around gathering money for a tzedoko also in a time when it is forbidden to interrupt is not a good thing because in so doing, they (lit. annul) disturb those davening from their prayers and their kavanah (intention).
In note dalet of the 'dvar halacha' section he elaborates by bringing the Mishnah Berurah 96:1 who quotes the Pri Megadim that the issue that Shulchan Aruch1 mentions (about not holding something - which includes money, during one's tefillah as it will leave the person distracted) is specifically during krias shema and pesukei d'zimra, and as such one shouldn't interrupt at this point.
Indeed Rav Chaim Kanievsky shlita in Derech Emunah, Matnas Aniyim, Perek 10 in the Tziyun HaHalacha 96 raises the point mention by @Shalom in the comments above:
ואפשר דאין חייבין ליתן להם דהעוסק במצוה פטור מן המצוה וכן באמצע התפילה
And it is possible that one is not obligated to give them since one who is occupied in a mitzvah is exempt from (another) mitzvah, and similarly (this applies when) in the middle of prayer.
1 It writes there:
כשהוא מתפלל לא יאחוז בידו תפילין ולא ספר מכתבי הקודש ולא קערה מלאה ולא סכין ומעות וככר מפני שלבו עליהם שלא יפלו ויטרד ותתבטל
When one is praying, one should not hold in one's hand tefillin, and not a book of holy writings, and not a full plate, and not a knife, or money, or a loaf [of bread] because one's heart is [focused] on them that they should not fall, and one will be distracted and will lose one's focus. (Sefaria translation)
Also note this idea in Gemara Brachos 23b - thanks @Kazi bácsi.