Would the conversion of a person, (knowledgeable in Halacha, meeting all the criteria for conversion), be halachically approved and functional,
if the Beit Din is formed of three observant Jewish sheidim?
Short answer: It's complicated, but I'm pretty sure it wouldn't be approved.1
Long answer: The core issue here, I believe, is whether or not a shed, even one termed "Jewish", can actually, halachically, be defined as a real Jew (on what exactly a Jewish shed is, see here, here and here).
A similar question is whether or not an alien can convert. The answer is inconclusive, in my understanding. While one answer brings Rabbi Aryeh Kaplan saying that while there may be aliens, they wouldn't have free will and as such could not convert, an answer on a different question brings a quote from Rabbi Soloveitchik in which he says:
It is possible that Hashem created other life forms on other planets. It is no problem to yahadus. The reason man likes to think he is the only created being in the entire universe is because of his egotistical nature.
Even the concept of am ha’nivchar may only be relative to our world, our small section of the universe. The Torah is written from the viewpoint of our sun, moon, and stars. It would not detract from our being the am ha’nivchar of this region of space if there were other am ha’nivchar in a distant galaxy. (The Rav Thinking Aloud, pg. 93)
In other words, theoretically there may be another species somewhere in the universe which has its own chosen nation and quite possibly, its own Torah (and there may even be more than one such other species). One of the core concepts in being a chosen people is the "Na'aseh v'Nishmah" - besides for Hashem having chosen us, we chose to accept the Torah.2 Further, Torah requires free will, for if there was no free will, why should one be punished or rewarded for doing something they had no control over? Therefore, it seems that according to Rabbi Soloveitchik, a theoretical alien chosen nation would have free will, too.
Now the question is, even if we go through the route of Rabbi Soloveitchik, are aliens truly comparable to shedim? I believe not. These aliens have free will. Ramchal in Derech Hashem 1:5:1 defines shedim as follows:
"ואולם נמצא מין א׳ של נבראים שהוא כמו אמצעי בין רוחני וגשמי, והיינו שבאמת אינו מורגש מחושינו, וגם אינו נגבל בכל גבולי הגשם המורגש וחוקיו, ומצד זה נקראהו שלא בדקדוק רוחני, אבל נבדל בענינו מן הסוג המלאכיי, אעפ״י שיתדמה לו באיזה בחינות, ויש לו חוקים פרטיים וגבולים מיוחדים כפי מציאותו באמת, ונקרא זה המין השידיי, שהוא מין השדים, ואולם גם הוא יתחלק לפרטים אחרים, שישוב המין הכללי סוג לגביהם והם מינים אליו."
Translation: "However, there is one species of created beings that is like an intermediate between the spiritual and the physical, meaning that it isn't truly sensed by our senses, and also isn't bounded by all the boundaries of the sensed physicality and its laws, and from this side of the issue we shall call it haphazardly "spiritual", but it is in its essence divided from the angelic type, even though it is similar to it in some aspects, and it has unique laws and special boundaries like its nature truly is, and it is called the shed species, which is the species of the shedim, though even it can be divided into different individuals, so that the common species will be a type upon them and they are the individual members of the species to it."
In other words, a shed is neither physical nor spiritual. Yet both are needed in a being to allow for free will. As such, it seems that shedim do not have free will3. Anything they do that appears to be them acting of their own free will is simply a semblance of free will. As such, it seems that they couldn't properly convert to Judaism. Per all of this, it seems that a shed could not convert.
However, Rabbi Shimon Chirari in Esmach BaHashem, siman 22 also mentioned this subject, with regards to the agadata that Ashmadai disguised himself as King Shlomo and ruled in his stead for awhile - does his being with Shlomo's wives invalidate them to Shlomo? It's a long tshuvah, so I'll only bring some of it and summarize the rest:
Rabbi Chirari begins by bringing the Ben Yehoyada on Gittin 68b who asks: If the biah of a shed is not allowed, why did Ashmadai do that with Shlomo's wives? After all, the Ari said: "שהיה יהודי ואם לא ישמור הלכות יהודאין למה נקרא יהודי?" (If he [Ashmadai] was Jewish, and he didn't keep the halachas of the Jews, why was he called a Jew?) - in other words, if he was Jewish, and his Judaism was on the condition that he keep halacha, why did he do something that was prohibited? From here we see evidence that the biah of a shed with a human woman isn't considered a biah, which is why a Jewish shed such as Ashmadai could do it without technically transgressing halacha.
On this Rabbi Chirari asks two questions:
- "מה שכתב אם לא שומר הלכות יהודאין למה יקרא יהודי וקשה וכי מי שחוטא לא נקרא יהודי חס ושלום והלא איתא במציעא נט אומרים לי דוד הבא על אשת איש מיתתו במה ואני אומר להם מיתתו בחנק ויש לו חלק לעולם הבא וכלל גדול בידינו אף על פי שחטא ישראל"
Translation: "What he wrote that if he doesn't keep the Jewish halachot why should he be called a Jew, and this is hard, for after all, is a person who sins not called a Jew ch"v? After all, it says in Bava Metzia 59 "they say to me: David, one who engages in intercourse with a married woman, his death is effected with what form of execution? And I say to them: One who engages in intercourse with a married woman before witnesses and with forewarning, his death is by strangulation, but he still has a share in the World-to-Come" and we have an important rule that a Jew, even if he sinned, is still a Jew."
- "ועוד שאלה אני שואל לדעת הגאון בן יהוידע שהשד היה שומר הלכות יהודאין איך תבע להו בנידותייהו וקשה לומר שגם זה מותר בשד עם הבת אדם"
Translation: "And another question I ask on the view of the gaon Ben Yehoyada that the shed kept the Jewish halachot, how could he demand of them, i.e., the queens, to engage in intercourse when they are menstruating? And it is difficult to say that this too is allowed with a shed and a female human..."
Rabbi Chirari expounds on the first question and explains that this issue means that the Ben Yehoyada's ruling isn't more conclusive than the Chida (also quoted by the Ben Yehoyada). On the second question, he says that most likely the biah of a shed isn't considered a prohibition, however the issue here is that this was a biah of a king (albeit, an imposter usurper in this case)4, which may have made the biah prohibited, but בדיעבד there was a leniency for the king's wives because of the threat of them becoming agunot.
In short, it seems that according to Rabbi Chirari, what makes a shed Jewish isn't exactly clear, for being a Jew goes beyond one's capacity in fulfilling halacha (and similarly, conversions aren't simply overturned because one stops keeping halacha).
I found this tshuva by Rabbi Refael Mazuz:
"בפשטות יש לומר שהם שני סוגים. יש שדים שנבראו בערב שבת בין השמשות (פרקי אבות פרק ה'), ויש שדים שנבראים מעוונותיו של אדם. וראיתי להגאון רבי יוסף חיים בבן יהוידע (גיטין ס"ח) שהביא מרבני האר"י שיש כמה מיני שדים: יש שדים תתאין והם רעין מאוד, ויש עליונים, ויש שדין נוכראין ויש שדים יהודאים, וכתב שם שאשמדאי הוא שד יהודי עיין שם. ולפי זה כנראה שגם השד שהיה במעשה דרשב"י (מעילה י"ז) הוא שד יהודי."
Translation: "In simple terms, we must say that there are two types [of shedim]. There are shedim who were created on erev Shabbat at dusk and there are shedim who are created from the sins of man. And I saw the gaon Rabbi Yosef Chaim in the Ben Yehoyada who brought from the rabbis of the Ari that there are a few types of shedim: There are low shedim and they are very evil, and there are high ones, and there are gentile shedim and Jewish shedim, and he wrote there that Ashmadai is a Jewish shed, see there. And per this, it seems that the shed in the story of Rashbi was a Jewish shed."
From here, it seems we can reconcile the two views: The first view, based on Rabbi Soloveitchik's answer, refers to the question of whether shedim can convert - it seems that they cannot, as they don't have free will. These are what can be termed as gentile shedim - they were created this way. However, there are other shedim who were created as Jewish shedim - from the sound of things, they are, in fact, inherently Jewish, as explain the Ari and Rabbi Chirari.
Bottom line, Jewish shedim are, apparently, inherently Jewish, though they may transgress halacha sometimes5 (as do most other people). However, they do not do this of their own free will, as they have none.
Now the question would be: Can shedim make up a beit din? This question hinges on the question of whether a shed can be a dayan at all? I believe not, as they can't make a proper decision on their own, for not having free will. Their actions are entirely controlled by external factors, including Hashem, which invokes things such as "we do not listen to a bat-kol", "lo bashamayim he" (the Torah is not in heaven), etc. As such, they cannot make rulings, cannot be dayanim and cannot make up any sort of court.
1 Yes, this is a lot of speculation and svarot as I was not able to find a direct source on the subject.
2 Setting aside the כפה עליהם הר כגיגית for a moment, though this can also be rectified with קיימו וקיבלוה in the time of Esther.
3 So also wrote Rabbi Eliyahu HaKohen in Agadat Eliyahu: "אלא אפילו רוחות ושדים שאינן בעלי בחירה ורצון וטבע מוטבע בהם להזיק בלי יראה ופחד עם כל זה בראותם התפילין כי שם ה' נקרא עליך כח השם מבטל הטבע המוטבע בהם להזיק שלא יזוקו" - Even dark spirits and shedim that don't have free choice and will and their nature is to cause harm without awe or fear, with all of this said, when they see the tefillin for the name of Hashem is called upon you, the power of the name cancels the nature embedded in them to cause harm and they do not harm.
And Rabbi Tzadok HaKohen Rabinowitz in Machshavot Charutz: "גבי שדים ...דליכא בהו הילוך מדרגות שיוכלו להתעלות מצד השתדלותם כי אינם בעלי בחירה" - On shedim...traveling between spiritual levels isn't relevant to them for they do not have free will.
4 Although it may be argued that he really was a king, for two reasons: a. Perhaps the din of a king depends on whether he is acting in a kingly position, even if he reached that position illegally (Ashmadai not being from the House of David) (The same might be argued about Ataliyah who was king's consort and later king's mother but really became queen (although I don't know if she was halachically queen; I don't know if there is such a concept in halacha) when she killed the rest of the royal family). b. Ashmadai, according to Yosef Sheda, is the king of shedim (though I don't know if that's a title that accounts for anything in halacha).
5 In the book "Ish MiBeit Lechem Yehudah" on Rabbi Yehudah Fatiya, a story is brought about a child in Rabbi Fatiya's community who claimed to have regular talks with Eliyahu HaNavi. Rabbi Fatiya figured out it was Jewish shed called Eliyahu who was tricking the kid, or in his words: "אין זה כי אם שד יהודי ששמו אליהו, ואינו אליהו הנביא, ואתה ניזוק משדין יהודאין." (This is nothing but a Jewish shed whose name is Eliyahu, and this is not Eliyahu HaNavi, and you are being harmed by Jewish shedim) - meaning that even if and when a Jewish shed sins, he is still called "Jewish".