Cain wasn't an idolator. Cain was chaos.
I can understand why you'd come to the conclusion (if he wasn't giving his best to Hashem then who was he giving it to?) but Cain himself actually possessed a symbolic character trait which permeated into the whole of human existence. Cain is chaos and disorder.
There's a wonderful article on Chabad which discusses the Legacy of Cain and how his character traits partner with that of Seth to become the standard model for all eras of humanity.
Cain is representative of chaos. Humanity as a whole undergoes Cainian volatility before settling down and establishing new order. Seth (the third sibbling) is representative of stability and order. His name literally means "To Establish" and represents the stable eras inbetween the eras of Cain.
Noah was a descendant of Seth and his wife Naamah was a descendant of Cain.
It's extremely symbolic that the new world would be built between the descendants of these two opposing figures. The idea is that it represents the legacy of both Cain and Seth and how humanity can embody the traits of both from era to era.
Cain wasn't an idol worshiper. Cain was a chaotic individual who embodied the self-destructive character traits that humanity continues to wrestle with. He is representative of the chaos that actively seeks to tear down what we establish in stability.