In B. Ketuvot 63a, we learn the following story (translation with commentary from R. Adin Steinsaltz):
רב יוסף בריה דרבא שדריה אבוהי לבי רב לקמיה דרב יוסף פסקו ליה שית שני כי הוה תלת שני מטא מעלי יומא דכפורי אמר איזיל ואיחזינהו לאינשי ביתי שמע אבוהי שקל מנא ונפק לאפיה אמר ליה זונתך נזכרת? איכא דאמרי אמר ליה יונתך נזכרת איטרוד לא מר איפסיק ולא מר איפסיק:
On the same subject it is related: Rav Yosef, son of Rava, was sent by his father to the study hall to learn before the great Sage Rav Yosef. They agreed that he should sit for six years in the study hall. When three years had passed, the eve of Yom Kippur arrived and he said: I will go and see the members of my household, meaning his wife. His father heard and took a weapon, as if he were going to war, and went to meet him. According to one version he said to him: Did you remember your mistress, as you are abandoning your studies to see a woman? (The Maharsha has the girsa of "your partner/זוגתך".) There are those who say that he said to him: Did you remember your dove? Since both father and son were involved in an argument, they were preoccupied and this Master did not eat the cessation meal before Yom Kippur and that Master also did not eat the cessation meal that day.
Isn't that a bit of an excessive reaction? A grown man, after spending three years of his life in learning, wishes to visit his home Erev Yom Kippur for one day. His father becomes infuriated, calls his son's wife a harlot, and tells him to go back to Yeshiva! Isn't it healthy for him to maintain a warm relationship with his spouse - how bad could it be for him to spend one afternoon with his household?
(See also B. Ketuvot 62b and Bereishit Rabba 95 for similar instances of very infrequent relationships between rabbinical colleagues and their spouses.)