There is a halacha that one may not name a child after an evildoer, a principle known as shem reshaim yirkav taken from Yoma 38b ("ועל בן קמצר וחביריו נאמר ושם רשעים ירקב", cf. Mishlei 10:7). Later authorities discuss the parameters of this halacha, of which you may see a summary here.
Now, occasionally there have been seeming exceptions to this rule, most famously ר' ישמעאל of the Talmud. A handful of other Jews have also been named ישמעאל -- including fairly recently (e.g. the late 18th century Rabbi Laudadio Sacerdote of Italy, whose Italian name is a calque of the Hebrew ר' ישמעאל כהן). There are various halachic justifications for the use of this name, but the very fact that it was used is assumed to be evidence that it is permissible (though not ever a popular choice among Jews).
The names Muhammad, Mahmud, Mehmet, and various similar names are extremely common throughout the Muslim world. Are there any examples of these being used as names by Jewish figures? If so, it would provide interesting evidence for a few discussions, including whether the Muslim figure Muhammad is viewed as a rasha, and whether names that are etymologically related to the name of a rasha are also forbidden for use.