This question already has an answer here:
Why is it that Jews from all parts of the world, do not look the same? We are, essentially, from the same forefathers, and how then does it seem that many Jews often look like nation that housed them in exile?
I once heard an answer that it is due to the constant "rape and pillage" the Jews were subjected to in their host countries, thereby marginalising the differences in appearance between Jew and gentile. But would that have such an extreme effect on the whole demographic of Jews? (e.g. most Yemenite Jews look Yemenite, and European Jews look European, so if only a few were subjected to rape, would the whole population look like them?). Is there any other answer to this question?