The Talmud, Pesachim 112a, says:
דאמר מר גרוש שנשא גרושה ארבע דעות במטה
When a divorcé marries a divorcée, there are four minds in the bed
Rashi explains that each participant is thinking about their first partner. This would seem to indicate that you never forget the people you were intimate with in previous relationships.
(thanks to mevaqesh for the source)