It discusses this in Eruvin (56a). The Gemara says
אליעזר איש ביריא אומר יושבי צריפין כיושבי קברים ועל בנותיהם הוא אומר (דברים כז, כא) ארור שוכב עם כל בהמה מאי טעמא עולא אמר שאין להן מרחצאות ורבי יוחנן אמר מפני שמרגישין זה לזה בטבילה מאי בינייהו איכא בינייהו נהרא דסמיך לביתא
The Gemara asks: What is the reason for this harsh statement with regard to the daughters of those who dwell in huts or travel in deserts? Ulla said: They do not have bathhouses, and therefore the men have to walk a significant distance in order to bathe. There is concern that while they are away their wives commit adultery, and that consequently their children are not really their own. And Rabbi Yoḥanan said: Because they sense when one another immerses. Similarly to the men, the women must walk a significant distance in order to immerse in a ritual bath. Since the settlement is very small and everyone knows when the women go to immerse, it is possible for an unscrupulous man to use this information to engage in adulterous relations with them by following them and taking advantage of the fact that they are alone.
This discusses the daughter but we can use the same idea to discuss the wife.
According to the second reason, the only concern is that that someone will stalk them and have relations against her will. While she would be permitted to a regular Jew, she would be forbidden to a Kohen.