שולחן ערוך יורה דעה · פז · יא
אם העמיד גבינה בעור קיבת כשרה יש בה טעם בשר אסורה ואם לאו מותרת אבל
המעמיד בעור קיבת נבילה וטריפה ובהמה טמאה אוסר בכל שהוא:
Paraphrase
Cheese made with kosher rennet is asur with נותן טעם (1 in 60), However rennet from neveilah, treifah, or a non-kosher animal is asur בכל שהוא (any minimal amount)
Your question as to why the halacha is this way, would seem to be the difference between something that is asur only by נותן טעם (which can be made batel - since it is intrinsically mutar) and something that is intrinsically asur even if it has no טעם (because even a mashehu has a major effect). Thus, the rennet, even though it might "no longer be a food" is still an asur item that has an effect on what it has been mixed with. This is not like the case in which a piece of neveilah meat falls into the soup pot (less than 1/60 and batel).
Note that this case is one in which the isur is added deliberately and it has not fallen in by accident. Thus the inyan of bitul le'chatchilah also applies.
As an analogy consider the arguments about gelatin which differentiate between gelatin from a kosher slaughtered animal and gelatin from a non-kosher or neveilah animal.(but that is a different subject).
Another point is that even if the neveilah rennet would have been batel beshishim, chaza"l made it asur by the takkana of gevinas akum. Thus, it would be asur now by that takkana even if a jew used it. For example, see the specified method explained by Kosher Cheese
It is too long to put in here, but I think that it is relevant.