The Mishnah says, in Yoma 2:1, “the assigner [of tasks] would tell [all of] them: "Stick out your fingers [for the procedure to assign the task]." And what would they stick out? Either one or two [fingers], and they would not stick out their thumb in the Temple.”
Why is it improper to stick out the thumb in the temple?
The Mishnah there is describing how certain jobs in the Basis Hamikdash would be appointed if there were multiple Kohanim vying for the same job. The Kohanim would form a circle around the supervisor, who would pick a large random number and a random position in the circle, and start counting the people around the line until he reached his number, and the person whom he landed on would get that job.
Each person was counted once and then the next person in line was counted, going around however many times until the random number was reached. However, since it is halachically forbidden to count Jews, the Kohanim couldn't be counted directly, so instead each Kohen would stick out a finger and that finger would be counted instead of them. Each Kohen was still only counted once, so they were only allowed to stick out a single finger (i.e. their index finger), or if it was too difficult to hold out one finger, they could hold out two fingers as one (i.e. holding out their index and middle finger out together as one).
Regardless of how many fingers they stuck out, each Kohen was only to be counted once in that round. However, if a Kohen saw that the count was close and if he would be able to trick the supervisor into counting him twice he would get selected, he might try to trick the supervisor by holding out both his thumb and his index finger so he gets counted twice instead of once. In order to prevent any of those shenanigans, a blanket rule was made banning using the thumb in the circle (this was only a concern regarding the index finger and thumb due to the distance between them, and not any other two non-thumb fingers).
This is based on the explanation of the Bartenura on that Mishnah.
I'm not coming to argue with the Bartenura who mentions the cheating aspect, the following is just a thought based on an admittedly inconclusive historical point.
It has been proposed that in Roman times sticking out ones thumb was a bad sign. Perhaps it was connected with a signal to kill a fallen gladiator. Perhaps only thumbs down, or up, or perhaps any direction the thumb sticking out was seen as bad, as opposed to a closed fist or open palm. See here and here for starters.
This is of particular note as Yoma, more than any other masechta, seems to spend quite a bit of time mentioning practices specifically from the end of the second temple, while the Jews were under Roman rule. The fact that an assumed common Roman insult was mentioned as being inappropriate in the mikdash would seem perfectly in line with the rest of the historical setting of the masechta.
From a kabbalistic perspective the thumb represents din or harsh judgement as opposed (!) to the other fingers. This is because the thumb has 2 joints or segments whereas the other fingers have 3 (the second middah is din, the third tiferet is a resolution of the tension between din and chesed/rachamim)
Perhaps this is why a thumb may not be exposed in the temple, as well as in other situations where one does not want to call upon the attribute of din, such as during the Amidah (Mishna Berurah 95:6), Havdalah (Magen Avraham 298:5 brings down the practice, sourcing the Zohar Vayakhel 208:1, Pri Eitz Chaim Shaar Shabbat chapter 24) and in general when one fears the evil eye (Berachot 55b)
