The Talmud, Pesachim 112a, says:
דאמר מר גרוש שנשא גרושה ארבע דעות במטה
When a divorcé marries a divorcée, there are four minds in the bed
Rashbam explains that each participant is thinking about their first partner:
בשעת תשמיש זה לבו לאשתו ראשונה וזאת לבעלה הראשון
This would seem to indicate that you never forget the people you were intimate with in previous relationships.