The Mishnah there is describing how certain jobs in the Basis Hamikdash would be appointed if there were multiple Kohanim vying for the same job. The Kohanim would form a circle around the supervisor, who would pick a large random number and a random position in the circle, and start counting the people around the line until he reached his number, and the person whom he landed on would get that job.
Each person was counted once and then the next person in line was counted, going around however many times until the random number was reached. However, since it is halachically forbidden to count Jews, the Kohanim couldn't be counted directly, so instead each Kohen would stick out a finger and that finger would be counted instead of them. Each Kohen was still only counted once, so they were only allowed to stick out a single finger (i.e. their index finger), or if it was too difficult to hold out one finger, they could hold out two fingers as one (i.e. holding out their index and middle finger out together as one).
Regardless of how many fingers they stuck out, each Kohen was only to be counted once in that round. However, if a Kohen saw that the count was close and if he would be able to trick the supervisor into counting him twice he would get selected, he might try to trick the supervisor by holding out both his thumb and his index finger so he gets counted twice instead of once. In order to prevent any of those shenanigans, a blanket rule was made banning using the thumb in the circle (this was only a concern regarding the index finger and thumb due to the distance between them, and not any other two non-thumb fingers).
This is based on the explanation of the Bartenura on that Mishnah.