The Talmud, [Pesachim 112a](http://hebrewbooks.org/shas.aspx?mesechta=4&daf=112&format=pdf), says:

> דאמר מר **גרוש שנשא גרושה ארבע דעות במטה**

> When a divorcé marries a divorcée, there are four minds in the bed

Rashbam explains that each participant is thinking about their first partner:

>  
בשעת תשמיש זה לבו לאשתו ראשונה וזאת לבעלה הראשון 

This would seem to indicate that you never forget the people you were intimate with in previous relationships. 

(thanks to [mevaqesh][1] for the source)


  [1]: http://judaism.stackexchange.com/users/8775/mevaqesh "mevaqesh"