A private mikvah used for baptism would be prohibited. As per BT Avodah Zarah p.58B and 59A, though, a body of water in the public domain cannot become prohibited, see discussion below:
ר"ל איקלע לבצרה חזא ישראל דקאכלי פירי דלא מעשרי ואסר להו חזא מיא דסגדי להו עובדי כוכבים ושתו ישראל ואסר להו
§ Reish Lakish happened to come to Bozrah, a town east of the Jordan. He saw Jews who were eating untithed produce and he deemed the produce prohibited to them. He also saw water to which gentiles bowed down and yet Jews drank the water, and he deemed the water prohibited to them.
אתא לקמיה דרבי יוחנן א"ל אדמקטורך עלך זיל הדר בצר לאו היינו בצרה ומים של רבים אין נאסרין
Reish Lakish came before Rabbi Yoḥanan and told him about the incident. Rabbi Yoḥanan said to Reish Lakish: While your coat [addemiktorakh] is still on you, go and retract your rulings. This is because the town of Bezer which is mentioned among the cities of refuge (Deuteronomy 4:43) is not the same as Bozrah. Bozrah is not part of Eretz Yisrael, and one is not obligated to separate tithes from its produce. The water is also permitted as it is water that belongs to the public, and water that belongs to the public is not rendered prohibited.
רבי יוחנן לטעמיה
The Gemara notes that Rabbi Yoḥanan conforms to his standard line of reasoning,
59a
דא"ר יוחנן משום ר"ש בן יהוצדק מים של רבים אין נאסרין הא דיחיד נאסרין
as Rabbi Yoḥanan says in the name of Rabbi Shimon ben Yehotzadak: Water that belongs to the public is not rendered prohibited. The Gemara infers that since water that belongs to the public is permitted, therefore, in a case where gentiles bow to water that is owned by an individual it is rendered prohibited.
ותיפוק ליה דהא מחוברין נינהו לא צריכא דתלשינהו גלא
The Gemara challenges: But Rabbi Yoḥanan could derive that even water owned by an individual is permitted, as the water is connected to the ground, and worshipping an object that is connected to the ground does not render it prohibited. The Gemara explains: No, it is necessary to derive this halakha from the fact that the water belongs to the public in a case where a wave raised the water and detached it from the ground. In this case worshipping water owned by an individual would render it prohibited.
סוף סוף אבני הר שנדלדלו נינהו תסתיים דר' יוחנן דאמר אסורות
The Gemara challenges: Nevertheless, the water ultimately falls into the category of objects that were detached without human involvement, such as boulders of a mountain that dislodged on their own. The Gemara (46a) cites a dispute between Rabbi Yoḥanan and the sons of Rabbi Ḥiyya with regard to boulders that dislodged without human involvement and were then worshipped, and does not conclude who deems the boulders permitted and who deems them prohibited. May it be concluded from Rabbi Yoḥanan’s statement that it is Rabbi Yoḥanan who says that the boulders are prohibited?
Text and translation Sefaria