The Ramban, Pirkei D'Rav Eliezer, amongst others say these were giants that fell from heaven and married common woman and had children that were giants like them.

The Pirkei D'Rav Eliezer says these were angels that fell from heaven and married the daughters of Kayin who were dressed inappropriately, and had children that were giants.

http://hebrewbooks.org/pdfpager.aspx?req=21858&st=&pgnum=131

http://hebrewbooks.org/pdfpager.aspx?req=21858&st=&pgnum=132