Suppose a Jew believes that there are multiple gods, but he does not worship them in any way. Is this a violation of the prohibition against idolatry for Jews?
We see from this answer that similar belief for gentiles does not constitute idolatry. Only the worship of other gods makes one guilty of idolatry. Is the same true for Jews, or does simple belief in the existance of multiple gods make a Jew idolatrous?
It is forbidden. I have included relevant snippets, but it's best if you read the whole thing.
Rambam Avodah Kochavim Chapter 2
Halacha 3
The worship of false gods is not the only subject to which we are forbidden to pay attention; rather, we are warned not to consider any thought which will cause us to uproot one of the fundamentals of the Torah. We should not turn our minds to these matters, think about them, or be drawn after the thoughts of our hearts.
....
What is implied? There are times when a person will stray after star worship, and times when he will wonder about God's oneness: Perhaps He is one, perhaps He is not?
...
This prohibition - though [severe,] causing a person to be prevented [from attaining a portion] in the world to come - is not punishable by lashes.
Halacha 6
Whoever accepts a false god as true, even when he does not actually worship it, disgraces and blasphemes [God's] glorious and awesome name.
The second Biur Halachah in the Mishna Berura (1:1), quoting the Sefer HaChinuch, discusses this question. It states clearly that a Jew is prohibited to believe in any power besides G-d. It additionally states that "even if someone admits that Hashem rules everything, but imagines that He gave over running of the world to an angel or star, he is considered one who accepts idol worship and transgresses "You shall not have any other gods"... rather he should believe that G-d himself and in His full glory oversees all the worlds and that no creation has any power to do anything except as He wills." Later, the Biur Halacha writes that the positive commandment of Shema is to know that, among other things, "Hashem is one without any "shituf" - anyone else at all."
I copied the relevant passage from Biur Halacha here:
ב) שלא נאמין בשום אלהים זולתו שנאמר לא יהיה לך אלהים אחרים על פני ואפילו מודה שהקב"ה שולט על הכל רק שידמה בדעתו שמסר הנהגת העולם למלאך או לכוכב ה"ז מודה בע"ז ועובר על לא יהיה לך אלהים אחרים על פני אלא יאמין שהקב"ה בעצמו ובכבודו משגיח בכל העולמות ואין לשום נברא כח לעשות דבר בלתי רצונו ולכן נקרא הקב"ה אלהי האלהים. ג) לייחדו שנא' שמע ישראל ה' אלהינו ה' אחד ופירושו שמע ישראל ודע כי ה' שהווה את הכל ברצונו והוא אלהינו המשגיח בכל העולמות הוא ה' אחד בלי שום שיתוף.
Pardon that I don't have a Rabbinic source to cite, but:
Shemot 20:3 You shall not have the gods of others in My presence.
This would appear to state that there exist the gods of others. So your question may be answered if by "belief" you mean something like 'an understanding that a thing exists'.
What it is that you 'believe' about those things that others call 'gods', and whether you have thus transgressed, is another matter.
Again, I wish I had better sources for the logic of these statements. Drawing from Ariel's answer to this question, citing Rambam on this exact topic, Halacha 3:
Since he may not know the guidelines with which to evaluate [ideas that will lead him] to the truth in its fullness, he may come to heresy.
I think the Rambam indicates that a person can have the capacity to make the evaluations [of this question] with right-discernment. I do not have more for this forum at this time, except to recommend that the Rambam has given a good guideline.
