This is a dispute recorded in Shulchan Aruch O.C. 8:14:
אם פשט טליתו אפילו היה דעתו לחזור ולהתעטף בו מיד צריך לברך כשיחזור ויתעטף בו הגה וי"א שאין מברכין אם יהיה דעתו לחזור ולהתעטף בו וי"א דוקא כשנשאר עליו טלית קטן והכי נוהגין
[R. Joseph Karo:] If he removed his tallit – even if his intent was to immediately put it back on – he needs to make a ...
If one removes the tallit or tzitzit (for instance, if one needs to
use the restroom during prayers), with the intention of donning the
same tallit or tzitzit afterwards, then it is not necessary to recite
the blessing again when putting it back on.
Baruch Hashem, I'm not alone. This question is addressed by various sources, all brought in Ma Shehaya Hu Sheyihyeh Shir HaShirim II 5:3 fn. 8 by Rabbi Eliyahu Wolf from Yeshivas Ner Yisroel, Baltimore. He brings as follows:
Birkas Aharon Berachos § 268 by Rav Aharon Levine asks this question, as well on how parshas Kedoshim ends. He leaves it as צ"ע.