Forgive me for a short preamble, but it is necessary to establish what scientists (i.e. practitioners of science) mean with ‘unscientific’ before exploring how it differs from what believers of scientism (i.e. science as a worldview, excluding all other worldviews) believe, and whether that could be avoda zarah.
Firstly, this question conflates the scientific method, as a tool, with the worldview that elevates the conclusions drawn with that tool beyond its reach. Science as a concept is just a tool: a social contract between those that decided to find and explain patterns in the world. The social contract governs the criteria that constitute what the scientific community would regard as adequate proof for any given claim.
When scientists claim that something is unscientific it does not necessarily follow that the claim therefore must not be true. An unscientific question or claim is just something that cannot be proven or disproven using the currently agreed upon scientific tools. It is beyond the realm of what science can believably claim, and so does not further our understanding within this social contract. Any claim that cannot be tested within this framework is deemed unscientific, but note that this says nothing about the validity of the claim.
For instance: to state that no purple polar bears exist is an unscientific question, because it is impossible to measure 'none-existence'; one simply never knows whether all the polar bears in existence have been found and their color established. There may very well be a purple polar bear, hiding from scientists in a cave somewhere: absence of evidence is not evidence of absence. In that regard, science itself cannot be idolatry: the metaphysical simply falls outside of what scientists can form hypotheses and scientific proofs on. To ask whether a deity exists, is an unscientific question, but the tool to establish that ultimately does not impact the reality. When people say that the Egyptians were wrong, because we can see the Sun and no beetle rolls that Sun in front of the Earth daily, the only scientific reply would be: have you looked in all possible dimensions? Our ability to perceive it (or measure it with current scientific tools) does not change the reality; just our ability to provide the agreed upon proof.
To conclude: science itself is a set of rules which changes every once in a while; those rules are an agreed upon set of criteria to judge the evidence to a particular claim.
Now that the preamble is over, it becomes possible to distinguish between the users of the scientific tool in scientific discourse and the believers of scientism who hold the view that “ the characteristic inductive methods of the natural sciences are the only source of genuine factual knowledge and, in particular, that they alone can yield true knowledge about man and society". When is this a proper application of the scientific method, and when does it start to veer into the realm of “if I see it, I will believe it”. Or in other words still: the difference between “we can finally agree upon the proof about a claim” and “something is not true, unless it has been proven”.
In this divide I believe important to point out that none of the big name philosophers that have been connected with scientism (e.g. Popper, Weber) would argue that something is only true if proven). However, inside and outside of the scientific community there are those that go beyond the scientific method and who do make such claims. So let me establish that in and of itself scientism does not have to be either avadah zarah, or even problematic from a jewish perspective. The question is rather; can those who elevate science to the realm of religion (i.e. the only fundamental universal truth) veer into the realm of avodah zara?
I don’t consider myself able to conclude whether someone’s belief constitutes avodah zara (chas vechalila), there are learned people who are infinitely better able to establish this than I. I can just share my thoughts.
The RAMBAM in mishneh torah makes an interesting point about how a thought process can start with God and the instruments He uses, and end with worshipping the tool
בימי אנוש טעו בני האדם טעות גדול ונבערה עצת חכמי אותו הדור ואנוש עצמו
מן הטועים היה וזו היתה טעותם אמרו הואיל והאלהים ברא כוכבים אלו וגלגלים
להנהיג את העולם ונתנם במרום וחלק להם כבוד והם שמשים המשמשים לפניו
ראויין הם לשבחם ולפארם ולחלוק להם כבוד וזהו רצון האל ברוך הוא לגדל
ולכבד מי שגדלו וכבדו כמו שהמלך רוצה לכבד העומדים לפניו וזהו כבודו של
מלך כיון שעלה דבר זה על לבם התחילו לבנות לכוכבים היכלות ולהקריב להן
קרבנות ולשבחם ולפארם בדברים ולהשתחוות למולם כדי להשיג רצון הבורא בדעתם
הרעה וזה היה עיקר עבודת כוכבים וכך היו אומרים עובדיה היודעים עיקרה לא
שהן אומרים שאין שם אלוה אלא כוכב זה הוא שירמיהו אומר מי לא ייראך מלך
הגוים כי לך יאתה כי בכל חכמי הגוים ובכל מלכותם מאין כמוך ובאחת יבערו
ויכסלו מוסר הבלים עץ הוא כלומר הכל יודעים שאתה הוא לבדך אבל טעותם
וכסילותם שמדמים שזה ההבל רצונך הוא:
During the times of Enosh, mankind made a great mistake, and the wise
men of that generation gave thoughtless counsel. Enosh himself was one
of those who erred. Their mistake was as follows: They said God
created stars and spheres with which to control the world. He placed
them on high and treated them with honor, making them servants who
minister before Him. Accordingly, it is fitting to praise and glorify
them and to treat them with honor. [They perceived] this to be the
will of God, blessed be He, that they magnify and honor those whom He
magnified and honored, just as a king desires that the servants who
stand before him be honored. Indeed, doing so is an expression of
honor to the king. After conceiving of this notion, they began to
construct temples to the stars and offer sacrifices to them. They
would praise and glorify them with words, and prostrate themselves
before them, because by doing so, they would - according to their
false conception - be fulfilling the will of God. This was the essence
of the worship of false gods, and this was the rationale of those who
worshiped them. They would not say that there is no other god except
for this star. This message was conveyed by Jeremiah, who declared
(10:7-8): "Who will not fear You, King of the nations, for to You it
is fitting. Among all the wise men of the nations and in all their
kingdoms, there is none like You. They have one foolish and senseless
[notion. They conceive of their] empty teachings as wood;" i.e., all
know that You alone are God. Their foolish error consists of
conceiving of this emptiness as Your will.
And we have also learned shemot 20:3
לֹא יִהְיֶה לְךָ אֱלֹהִים אֲחֵרִים עַל פָּנַי: You shall not have the gods
of others before me
Which rashi explains:
the gods of others: which are not gods, but that others have made them
for gods over themselves. It is impossible to interpret this passage
to mean: gods other than I, since it is a disgrace for Heaven to call
them gods along with Him. Alternatively: strange gods, for they are
strange to their worshippers. They cry out to them, but they do not
answer them, and it appears as if it [the god] were a stranger, who
never knew him [the worshipper].
When an individual starts elevating G’ds tools (the universal laws of nature) to the only true gods. Avadah Zarah does not need an actual deity; it seems sufficiently troubling to take anything and make it a god for yourself. So what can be elevated to the level that it comes before G’d in [extremist] scientism? When man’s own intellect and what it is currently able to comprehend and prove is stated to be the only factual truth in the universe, that person is not only unscientific, but has started to elevate its own intellect above everything else. In this worldview, the individual intellect comes before G’d: because only as long as the intellect decides to accept it, G’d will exist. A subjective and conditional existence of G’d places oneself before G’d in such a worldview. ("G’d only exists if I decide He exists"). I would argue that such an extreme position could be regarded as a form of idol worship, with the above texts in mind.
(who worships his own mind, i.e. he elevates the human mind as some kind of idol since only that which he can demonstrate and prove with his mind exists
This sounds like a mussar lecture, rather than a halakha lecture. Placing high value on one way of finding truth has nothing to do with the halakhic parameters of idolatry TTBOMK.