My answer will probably be a bit scattered to begin with as this is such a broad topic and i will probably refine it as people comment and as i'm able to find more definitive sources.
A "cultural Jew" in an American context has come to mean quite a few things, it could be someone who was born into a Jewish family, was raised with certain customs and maybe was even educated in Judaism, but came to disagree with the religion and therefore considers himself a cultural Jew in that he rejects the religion but accepts the ancestry. It can also refer to someone that was born into a family of Jews that was or is ignorant of the religion and was never taught anything of the religion, but certain cultural things were still passed down, such as food, ways of thinking, or outlooks on life. A man who grew up speaking Yiddish an eating Gefilte Fish but never attended a Synagogue or learned Hebrew is still as "culturally Jewish" as one who went through the Yeshivah system and walked away from it.
Food is often a big determining factor of a cultural Jew as food is always a defining character of ethnicity. How do i know when i'm at an Egyptian Jew's house as opposed to an Irqi or Syrian Jew's house? Only the Egyptian Jews prize a food called Molokhia/Molokheya. Think of the difference between Koreans, Japanese and Chinese. You can say that they all have different languages, histories, and heritages, but when they are raised in America and have only been taught language, often their most distinguishing feature is the foods that they grow up with and feel partial to.
This idea of a cultural Jew is much more heavily influenced from Jews reigning from Europe, as European Jewry became much more sectarian and defined by specific ideas, beliefs, or ways of living. An observant Jew looks/dresses different(ly) than a nonreligious Jew, whereas in some places of the middle east, this kind of distinction wouldn't have been as obvious amongst Jewry there. And while these undercurrents of nonreligious Jews who still considered themselves Jewish started as a natural Jewish phenomena, the entire process was quickened and solidified by Hitler and his attempt to exterminate the Jewish people. Because to Hitler, it didn't matter if you were an actively religious Jew, or if you happened to have one great grandfather on your father's side that was a Jew, you were all going to the concentration camp. And so suddenly, the heavily atheistic "educated" Jews of Europe were in the same train cars next to Chassidic Rabbis from rural villages, with everyone being stripped down and reduced to the same thing, a number set in motion toward death.
When Israel became a nation they often defined who is "a Jew" based on Hitler's definitions of who a Jew was or wasn't. Since the adage of Israel was "Never Again," and was designed as a safe haven, a place for Jews to flee, they had to struggle with the idea of "who is a Jew?" Is it someone that's halakhically considered a Jew? Is it someone who is currently religious? Or is it anyone facing persecution for being Jewish regardless of religious observance or lineage? Israel has often accepted the last view to be their definition, which has caused Rabbis to ask, should Hitler be the one who decides who is or isn't a Jew? Should White Supremacists who bomb reform or reconstructionist Temples be the ones who decide who is Jewish? There are no definitive answers to these questions, but these questions have led to an entire new category of Jewry, the cultural Jews.
This category of Jews now that it has become firmly established will continue to grow and change through time, but it has now become cemented as an entity that will continue to persist and define itself.