Rambam (Hil. Yesodei Hatorah ch. 7) describes the prerequisites that a person needs in order to be a navi. To put it in contemporary terms, these are needed so that his mind is "tuned to the correct frequency" on which to receive Hashem's word. The average person has not reached this level - they would no more be able to receive such a transmission than a toaster can receive an FM broadcast.
I've also seen an idea somewhat related to this (whose source I don't recall), that prophecy requires the receiver's personality to be effaced as much as possible, so that the message doesn't get distorted. As it is, with most prophets (except Moshe, as the Rambam explains there), it does inevitably get filtered through their own subjectivity (this is why the Gemara, Sanhedrin 89a, states that "no two prophets use the exact same terminology" - they may both receive the same message from Hashem, but each one perceives it slightly differently), but that has to be kept to a minimum. According to this approach, too, for Hashem to speak to the average person would be a purposeless exercise: the message would be so thoroughly mangled in its passage through his or her mind as to be unrecognizable.