I understand that, before World War Ⅱ, the Jews of Hungary and Germany mostly spoke Hungarian and German, respectively, amongst themselves (at home), whereas the Jews of Poland mostly spoke Yiddish amongst themselves, not Polish.
Many Jews spoke Hungarian in Hungary because there was a very successful policy of Magyarization in Hungary. This is one of the explanations for the rise of ultra-orthodoxy in northeast Hungary (the 'Unterland'), and the invention of a new Halakhic tradition under the disciples of the Hatam Sofer (d. 1839), as a reaction against the great transformation of society in Hungary in the 1860s--economic, religious, national, and linguistic.
Maybe the Poles in the Jewish areas were generally lower-class peasants with whom the Jews had little interest in culturally assimilating? I recall a story about Rav Boruch Ber Leibowitz zt"l, where he greeted a man working in his house with "Good Morning" in Polish, and then apologized profusely after realizing that the man was actually Jewish. Apparently it was considered somewhat demeaning for a Jew to be addressed in Polish rather than Yiddish. Though it's quite possible that Rav Boruch Ber would have acted the same with any 'goyishe' language, not specifically Polish.
Edit: I found the following version of this story here:
R' Boruch Ber once saw a man working in his house, and greeted him in Polish. The man laughed at him. R' Boruch Ber asked his rebbitzen to find out why the man was laughing at him. The workman answered, "The rov thought I was a ‘goy’, and he greeted me in Polish. I am a ‘yid’, and he could talk to me in yiddish." When R' Boruch Ber heard this he turned pale. Oy! I must ask the man for mechila. He approached the man and begged his forgiveness. The man laughed it off, but R' Boruch Ber persisted.