Two incidents described in the Talmud there might give us a clue:
ההוא דנחית קמיה דרבה ואמר אתה חסת על קן צפור אתה חוס ורחם עלינו אמר רבה כמה ידע האי צורבא מרבנן לרצויי למריה א"ל אביי והא משתקין אותו תנן ורבה נמי לחדודי לאביי הוא דבעי ההוא דנחית קמיה דר' חנינא אמר האל הגדול הגבור והנורא והאדיר והעזוז והיראוי החזק והאמיץ והודאי והנכבד המתין לו עד דסיים כי סיים א"ל סיימתינהו לכולהו שבחי דמרך למה לי כולי האי אנן הני תלת דאמרינן אי לאו דאמרינהו משה רבינו באורייתא ואתו אנשי כנסת הגדולה ותקנינהו בתפלה לא הוינן יכולין למימר להו ואת אמרת כולי האי ואזלת משל למלך בשר ודם שהיו לו אלף אלפים דינרי זהב והיו מקלסין אותו בשל כסף והלא גנאי הוא לו
A certain [reader] went down [before the Ark] in the presence of Rabbah and said, 'Thou hast shown mercy to the bird's nest, show Thou pity and mercy to us'. Said Rabbah: How well this student knows how to placate his Master! Said Abaye to him: But we have learnt, HE IS SILENCED? — Rabbah too acted thus only to test Abaye.
A certain [reader] went down in the presence of R. Hanina and said, O God, the great, mighty, terrible, majestic, powerful, awful, strong, fearless, sure and honoured. He waited till he had finished, and when he had finished he said to him, Have you concluded all the praise of your Master? Why do we want all this? Even with these three that we do say, had not Moses our Master mentioned them in the Law and had not the Men of the Great Synagogue come and inserted them in the Tefillah, we should not have been able to mention them, and you say all these and still go on! It is as if an earthly king had a million denarii of gold, and someone praised him as possessing silver ones. Would it not be an insult to him?
(Soncino translation)
In the first incident someone violated one of the rules mentioned in the Mishnah for which we silence people. Yet it does not say that Rabbah, or Abbaye, interrupted the person or threw him down. It appear that they merely raised the issue afterwards.
In the second incident we are introduced to another problematic form of prayer – using too many adjectives to describe God's greatness. In this case, too, we do not find that R. Chanina interrupted the person. In fact the Talmud explicitly states He waited till he had finished.
In his commentary there, R. Samuel Eidels point this out and suggests that that is in fact what the Mishnah meant by "we silence him":
הא דלא היו משתקין אותו בתפלתו לא רבה ולא אביי י"ל דלאחר תפלתו קאמר דמשתקין אותו שלא יעשה כן פעם אחרת וכן לקמן גבי המרבה בשבחות קאמר המתין לו עד דסיים כו
As for this that neither Rabbah nor Abbaye silenced him during his prayer, one could say that [the Mishnah] meant that after his prayer we silence him so that he won't do it again. And similarly, later on by the one who was excessive with praises it says that he waited for him until he finished, etc.
On the other hand, R. Ezekiel Landau in his commentary there interprets "silence him" literally:
לשון הרמב"ם בפירוש המשנה ענין מה שאמרו על קן צפור וכו' שיאמר כמו שחמלת על קן צפור וכו' כן רחם עלינו וכן פירש הברטנורא וכבר ראיתי מי שתמה הלא סיום דבר זה לא נזכר במשנה אבל טעות הוא בידם שהתנא קמ"ל שאין מניחין אותו לסיים דבריו ומשתקין אותו תיכף בתחלת דבריו
The wording of the Rambam in the Commentary to the Mishnah is the matter of that which they said "on the bird's nest etc." that someone says "just like you had mercy on the bird's nest etc. so too have mercy on us. And the Bartenura explains likewise. And I have already seen people who wondered "this ending is not mentioned in the Mishnah!" But it is a mistake in their hands, because the tanna is teaching us that we don't leave him to finish his words; we silence him immediately at the beginning of his words.
A Gaonic interpretation (Otzar Hagaonim, Berachot, HaPeirushim p. 46) refers to it as removing the person with scolding:
עד דאמ' כזה מסלקין אותו בנזיפה
R. Moses Shick in a responsum (O.C. 31) cites the questioner as saying:
וגם מה שאמרו בברכות [ל"ג ע"ב] דשמע שמע כגון דאמר מילתא מילתא ותני לה מגונה הוי ואף דאין משתקין אותו בשעת מעשה מכל מקום צריך למחות בו שלא יעשה כן בפעם אחרת כיון דהוי מגונה
And also that which they said in Berachot (33b) that "shema shema" in the form of saying it word by word, they taught about it that it is deplorable, and even though we don't silence him at the moment we nevertheless must protest it so that he doesn't do it again, since it is deplorable.
And then acknowledges that the questioner is correct:
ונכונים כל דבריו
And all his words are correct.
If for things which the Talmud does not rule to "silence him" we don't silence him in the moment but instead rebuke him later, the implication is that when the Talmud does rule to "silence him" we silence him on the spot.
