It says in Shulchan Aruch Evan HaEzer Siman 21 Sif 1 that it's ossur to look at the colored clothing of a woman that you know even if she is not wearing them, lest one come to "hirhurim" (to think about) that woman.
What about if someone's wife borrows a dress from her friend that you know, is it a problem then to look at one's wife when she is wearing that dress (or to specifically look at the dress.)