These are great questions that I hope I don't ruin by answering, but indeed there are answers in the sources. The first stop we take is to prove that indeed there is an end to the whys. Hashem's ratzon is absolute, and therefore what He wants, He wants for it and not for a secondary reason. This is brought in the Guide to the Perplexed 3:13:
יהיו ... הנמצאות כולם מכוונות לעצמם לא מפני דבר אחר
Each and every being exists for its own sake, and not because of some
other thing
Rabbi Manis Friedman, after his own teachers, puts it rather bluntly1: if you ask Hashem why He wants a particular mitzvah, He will tell you He doesn't know! It's not a "knowledge" thing, it's on a level altogether above that.
Next, the list you have given is not a list of equals. The Dira Betachton reason, and the desire to have a relationship are actually one and the same (and intertwined with the notion of wanting to do good to His other, too!), and this is the highest reason, and actually is relevant to Him Himself (see the midrashim this is taught from2). When the sources ask why He wants that, they conclude the same as above, that He wants it not to solve a higher problem; that is what He wants. That's why Chazal call it a ta'avah.
The reason we can't ask why on that is because there is no why. Or rather, the only answer you can give is "because that's Him". If you try to play the why game here, you hit a genuine dead end; why is He the way He is and not another way is a nonsense (and probably forbidden) question.
You should note that it was explained in 1952 in a maamer called "B'sha'a Shehikdimu"3 by the Lubavitcher Rebbe that there is a hidden inner reason we learn from the Kabbalah as to why He wants a relationship, and the answer is, He wants a relationship with Nishmat Yisrael specifically. To speak in the way of man, He never wanted to get married until He met her. So truly, the "reason" for creation is at least as elusive, and infinitely more so still, than trying to ask a man why he married his wife. Going to the world of reason to find the answer is inappropriate, and insensitive.
This is a very beautiful and Divine way to have a relationship; to tell one's spouse that there's no reason for their relationship. He doesn't want us in order to achieve something or because of something about us. He just wants us, for us, because it's us.
This answers one of your smaller questions. Why does doing good require the good to be earned and the Ramchal gives our answer in Derech Hashem 1:2:
הנה איננו שלימות אמיתי אלא יקרא שלימות בערך אל ענין חסר ממנו אך
השלימות בהחלט אינו אלא שלימותו ית׳ וע״כ בהיות חפצו ית׳ להטיב לזולתו
לא יספיק לו בהיותו מטיב קצת טוב אלא בהיותו מטיב תכלית הטוב שאפשר
לברואים שיקבלו ובהיותו הוא לבדו ית׳ הטוב האמיתי לא יסתפק חפצו הטוב
אלא בהיותו מהנה לזולתו בטוב ההוא עצמו שהוא בו ית׳ מצד עצמו שהוא הטוב
השלם והאמיתי
And therefore since His desire, may He be blessed, was to do good to
others, it would not be sufficient for Him to do a little good, but
rather [only] in giving the full goodness that is possible for the
creatures to receive. And in His alone, may He be blessed, being the
true good, His good desire would only be satisfied by giving others
that very good that is within Him, may He be blessed, from the angle
of Himself - which is the truly perfect good...
כי אלו היה מוכרח במעשיו להיות בוחר עכ״פ בשלימות לא היה נקרא באמת בעל
שלימותו כי איננו בעליו כיון שהוכרח מאחר לקנות והמקנהו הוא בעל שלימותו
ולא היתה הכונה העליונה מתקיימת
...For if he were forced in his actions to choose perfection no matter
what, he would not truthfully be called the master of his perfection;
as since he is forced to acquire [it] by another, he is not its
master. [Rather] it is the one who makes him acquire it that is the
master of his perfection. So [God's] intention would not have been
fulfilled.
i.e. it's because that's the way He is; master of His own good, therefore to give us that level of good we have to be like Him and master our own good. To put it in terms of relationships, in order to be in a holy relationship, there must be consent as well as an opportunity to truly express ourselves in our love and connection with our God (by making the Dira for Him and inviting Him in ourselves). That's what He wants, and not robots. Why? That's the way He is. It's not so shocking either, we can see the benefit of consent and true freedom in a Divine relationship, even if we don't have enough words to express it.
Finally, the more rational reason given about being a King in need of a people is more a secondary reason, and it is explained in Sha'ar HaEmuna VeYesod HaChassidut:
From the point of view of the creation, it seems that if it was God’s
wish to be King, then the only way He could have achieved this goal
was to create a world over which He would rule, for there is no king
without a people. Yet, this very notion is a created understanding.
From the point of view of God’s Sovereignty, there really is a King
without a people, who can reign before any of the creation was formed,
since God is all-powerful (from the great summary by translator R'
Betzalel Edwards)
(something I also heard, give or take, from R YY Jacobson) which is that He doesn't really need a people to be King, and that's only something true from our perspective, but not from His. Our rational brains are limited to a certain paradigm due to His decree, which will be modified in future.
1 - See the shiur "When God Makes No Sense", for example.
2- Bamidbar Rabbah 13:6 , Midrash Tanchuma Bechukotai 3:1, Midrash Tanchuma Nasso 16:1. Seem commentaries on founding pasuk Shemot 25:8
3- בשעה שהקדימו, יום ב׳ דחג השבועות תשי״ב