The procedure you are mentioning is hatarat nidui from the heavens.
That means that if a person got a nidui from above, he can ask 10 men to cancel it. Since among each 10 men there is shechina, when they cancel the nidui, the heavens have to accept this.
This custom is mentioned in this booklet of minhagim, in the end of page 24, more details in footnote 119.
The source is the Gemora in Nedorim 8a:
אמר רב יוסף נידוהו בחלום צריך י' בני אדם להתיר לו
Rav Yosef says, someone who got nidui during a dream, needs 10 men to cancel it
Since we do not remember all our dreams, there is a custom to make this, just in case.
A regular Herem of beis din can not be removed by this way.