Is it considered a hefsek to play with one's fingers or silverware or such while waiting for the challah to be passed around on shabbat, after the blessing made on it? Even though it isn't talking, it is an activity unrelated to eating bread.
I don't have any sources off the top of my head to corroborate this, but I would say it's not a problem. The whole problem with talking in between making the bracha and the concluding act of eating is that you are taking the attention away from the main activity of the moment - waiting for the bracha to be said. Although playing with your fingers is not related to this, as long as it's not disruptive, it should have the same status as adjusting your seat when you sit down at the table.