Ramban makes this observation (Deut. 33:1), and says "anyone with a good intellect" can figure out the reason. Okay I'm stumped ... help me out here please?
Maybe it's related to the idea that the Name Havayah refers to Hashem as He transcends all of creation, while Elokim represents His immanence within it. Describing someone as ish ha- implies a closeness with the linked object (as in this coming week's parshah, where Noach is described as ish ha-adamah, "a man of the ground," i.e., someone devoted to farming - or as in the common use of "ish" to mean "husband"). But since "no man may see Me and live" (Ex. 33:20), then it's impossible to describe any human being as "ish Havayah."