The reason he gave was that the Zionists placed a high priority on physical health and did so by playing ball.My response to this is, "The Zionists placed a high priority on getting places quickly and on time, so we should assur cars to not be like the Zionists." Is there any actually halachic justification for this seemingly arbitrary minhag (for lack of a better word)?
This is a very common question asked on many things that people do not do because it is considered non-Jewish/Chukas Hahukum which is prohibited in the Torah. (V'Yikra 20:23)וְלֹ֤א תֵֽלְכוּ֙ בְּחֻקֹּ֣ת הַגּ֔וֹי People frequently say "non-jews drive cars and wear pants so by that logic yo:u should prohibit them as well"
The Rishonim give the following distinction for what is(n't) considered chukas hakum. Something that non-Jews innovated or do for practical reasons is not considered chukas hakum. Something that non-Jews do or enjoy for cultural reasons is considered chukas hakum. Having seen and heard this topic discussed by Rabbis ranging from Satmar to YU they all seemed to accept this distinction. The only question is it's practical application. In Satmar there are far more things that they consider to be cultural rather than practical than they do in YU so there are far more things Satmar opposes for such reasons (including playing ball in the US) than they do in YU. Both agree however that imitating non-Jewish culture is an halachic problem
Getting back to the original question if playing ball is perceived as a part of Zionist culture than there would be a reason to prohibit it. There is nothing cultural about wanting to getting places quickly and on time. It is a universally high demand practical issue. Therefore no one opposes cars even if Zionists are into them.
Note: The halachos of chukas hakum are far more complicated than written above. It's not just one simple distinction.