Maimonides describes the qualifications of a Jewish prophet. He also describes how to discern a prophet who appears to meet the qualifications, but still is shown to not be an authentic prophet. Among them:
Therefore, if a prophet arises and attempts to dispute Moses' prophecy by performing great signs and wonders, we should not listen to him. We know with certainty that he performed those signs through magic or sorcery. [This conclusion is reached] because the prophecy of Moses, our teacher, is not dependent on wonders, so that we could compare these wonders, one against the other. Rather we saw and heard with our own eyes and ears as he did.
He goes on in the next chapter to describe something specific that would dispute Moses' prophecy:
Similarly, if [a "prophet"] nullifies a concept which was transmitted by the oral tradition, or states with regard to one of the Torah's laws that God commanded him to render such and such a judgment, or that such and such is the law regarding a particular instance and the decision follows a certain opinion, he is a false prophet and should be [executed by] strangulation. [This applies] even if he performs a wonder, for he is coming to deny the Torah, which states: "It is not in the heavens."
As far as I know, one of the claims of the Muslim religion (or perhaps the Koran itself?) is that the correctness of the texts of the Jewish Bible is disputed. This contradicts the idea of what is "transmitted by the oral tradition" at the most fundamental level.