The prevalent opinion (meyikar hadin) is that it is permitted for men to wear a wristwatch and it doesn’t violate “lo yilbash” (prohibition against wearing opposite gender’s garments), it is only a stringency to refrain from doing so.
לא תעדה אשה עדי האיש כגון שתשים בראשה מצנפת או כובע או תלבש שריון וכיוצא בו (ממלבושי האיש לפי מנהג המקום ההוא) (טור) או שתגלח ראשה כאיש ולא יעדה איש עדי אשה כגון שילבש בגדי צבעונים וחלי זהב במקום שאין לובשין אותם הכלים ואין משימין אותו החלי אלא נשים: הגה ואפילו באחד מן הבגדים אסור אף על פי שניכרים בשאר בגדיהם שהוא איש או אשה (ב"י) טומטום ואנדרוגינוס אסורים להתעטף כאשה:
A woman may not clothe herself in men's clothing, e.g. put on her head a mitre or helment, or wear armor, and so on (Rama: from that which men wear in accordance with the local custom (Tur)) or shave her head like a man. A man may not clothe himself in the clothes of a women, e.g. wear colored clothing or golden ornaments in a place where only women wear those things. (Rama: Even wearing just one of the garments is forbidden, even if it is apparent by his other garments that this is a man or a woman (Beis Yosef).) A tumtum and an androgynos are forbidden to dress like women.
It is widely know that HaGaon HaRav Chaim Kanievsky z”l was of the opinion that wearing a wristwatch violates “lo yilbash”.
Rav Chaim z”l gave many people a bracha and advice when they visited with him seeking just that. The Rav also frequently requested that they accept upon themselves something to uplift themselves spiritually, such as growing a beard, peyos, and not to wear a wristwatch to avoid “לא ילבש”. He points out that the Chazon Ish and his father, the Steipler, did not wear wristwatches because of the prohibition.
Rav Kanievsky z”l was shown photos in which Rav Shach z”l and Rav Elyashiv z”l were wearing watches. The Rav reportedly responded explaining that he too used to wear one until he received reliable eidus from Maran Chazon Ish that it is prohibited.
The Lubavitcher Rebbe also wore a watch before he took on the Nesius (leadership).
You can see the Rebbe wearing a wristwatch here: https://m.youtube.com/watch?v=qgcDbDnto1k
Chatzitza by Tefillin:
From the letter of the law, it is permitted to have a Chatzitza under the areas of the straps that are on the arm, and hence it is permitted to wrap the straps over a watch that one is wearing.
Nonetheless, it is initially proper to be stringent to remove the watch in order not to have a Chatzitza even under the straps.
Doveiv Meisharim 2:37; Cheshev Haeifod 1:21; Teshuvos Vehanhagos 2:26; Piskeiy Teshuvos 27:9