Rambam in Sefer HaMitzvos Lo Saaseh 115 writes
שבעת שחיטת הפסח והוא בין הערבים לא יהיה חמץ ברשותו לא אצל הזורק ולא אצל השוחט ולא אצל המקטיר ולא אצל אחד מבני החבורה, וכל מי מהם שיהיה חמץ אצלו בעת ההיא לוקה.
At the time of the slaughtering of the Pesach, meaning the afternoon, a person shouldn't have chametz in their possession. This applies to the one who slaughters it, throws its blood, burns its limbs, and all those who are part of this pesach's assigned group. If any of them own chametz at that time they get lashes.
There's a rule that's accepted as halacha that לאו שאין בו מעשה אין לוקין עליו
, a person doesn't receive lashes when they do a prohibition without an action. In this case, those involved with the temple service of the pesach are doing an action, so I understand why they get lashes. But the בני החבורה aren't doing anything, and the Rambam says they get lashes. Why is that?