Why did Hashem create things to work like they do?
Is Hashem describing a spiritual world, or the spiritual world, with his creation on earth? He could have created it in so many ways; why this way?
"Man in his own image". . . what about everything else?
Is everything in the universe a metaphor for a spiritual world? (If not then what? Because it looks nice?)
The Torah, sages, talmud must comment on this question in many ways.
This idea is explained at length and in great detail by Rabbi Yosef Gikatila in his book, Sefer Ginat Egoz (The Nut Garden).
The subject is illustrating what is meant from the Torah when it says that G-d 'said' the ten utterances.
He explains, following the teaching from Sefer Yetzirah, Chapter 4, Mishnah 1, that the 'Upper Worlds', meaning the spiritual realm, parallel the 'Lower World', meaning this physical world. This is a consequence of the 'Doubling Letters' of the Hebrew Aleph-Bet. They are know by the Acronym 'Beged Kaparat' (בג״ד כפר״ת). This acronym actually translates as 'A garment covers'.
The 'Degesh' in the letters indicates the 'hard sound' and alludes to the Sefirah of Gevurah and the concept of 'tzimtzum'. The absence of the 'Dagesh' indicates the 'soft sound' and alludes to the Sefirah of Chesed.
We are fortunate in this generation to have been exposed through mathematics and physics to the concept of the fractal, like with the Mandelbrot set. It illustrates for us that this very esoteric concept in Torah relating to the letters of the Torah serving as the blueprint for everything is actually demonstrated throughout all of creation.
The answer to this question depends on how one would interpret the concept of "Histakel Be'Oraisa Uvara Alma" - [Hashem] Gazed into the Torah and created the world.
Some interpret this phrase to imply that the Torah forms the "blueprint" for the world, encoded within it somehow. This does not intrinsically imply that the physical world is a metaphor for the spiritual one, only that the "rules and structure" of the physical world are "included" within the infinitude of Torah.
Conversely, there are those (a link mentioned in the comments above) who interpret this as saying that the physical world is a mirror or analog of a spiritual reality. There are a number of different facets to this philosophy, from the laws of physics existing as metaphor to the spiritual world to there being a "sympathetic relationship" between the spiritual and the physical (where a physical change modifies the spiritual world and vice versa).
So yes, there are a number of different viewpoints from the Sages regarding the relationship between the spiritual and physical worlds.
Regarding the second half of your question, there is no intrinsic requirement for such a (direct) relationship to exist - the physical world exists because Hashem wills it to. We inhabit a physical realm because working within it is our task.
In terms of why the physical world exists, this becomes a much more complicated side discussion, but I will touch upon one idea that addresses this. The Zohar talks about the principle of Nahama D'Kisufa, or "bread of embarrassment." Basically, the idea is that the receipt of an unearned "reward" demeans the recipient. Thus, though Hashem might "desire" to give boundlessly, to do so would be giving sub-optimally. It is therefore a GREATER act of giving by allowing us to earn the good we receive.
According to the Zohar, the world was created (and free will given) in order to furnish us with the capacity to earn our own reward, the greatest good possible. This would hold whether or not we understand the physical as a metaphor for the spiritual.
the universe is the way it is because this is the best way to fulfill its purpose. it's purpose is to be an arena when human free will can function as explained by the Ramchal in Derech Hashem. Thus everything is tailored around this. hence, it cannot be in many different ways but instead needs to be built around man's mission here.
some sources
shaar yichud ch.6
For to our senses and intellect it appears like a built and furnished house, whereby all its needs are prepared. The sky above like a roof, the land below like a carpet, the stars in their array like candles. All the objects gathered in it like provisions - everything has its need. Man is like the master of the house who uses all that is in it. The various types of plants are prepared for his benefit; the various kinds of animals serve his use, as David said: "You have made man to have dominion over the works of Your hands; You have put all things under his feet; All sheep and oxen, and the beasts of the field; The birds of the air, and the fish of the sea, and whatsoever passes through the paths of the seas" (Ps. 8:7)...and the purpose of everything is for the benefit of mankind, as Solomon said: "To every thing there is a season, and a time to every purpose under the heaven" (Eccles. 3:11)
and in the shaar bechina
You should know that what I have called your attention to in this treatise, is but a small portion of the vast knowledge concerning the mysteries of wisdom you can acquire by your own understanding-mysteries which will be revealed to you if you purify your heart and refine your soul. When you have attained in these matters the utmost knowledge of which you are capable, you should realize that all this knowledge which you have acquired of the Creator's wisdom and power, as manifested in this universe, is as nothing compared to His real power and wisdom. For what is manifested is only that which is needed for a human being, not according to the true extent of His power, for it is infinite. Hence you should think of the awe-inspiring nature of G-d and His infinite might as they essentially are, not as you with your limited intelligence can perceive them.
Ramchal Daat Tevunot Siman 40
certainly G-d could have established the world through His omnipotent system, in such a way that everything would be totally incomprehensible to us, not before not after, without cause and effect. If He had done so, no one (atheist) would be able to open their mouths for we would not be able to understand anything whatsoever... but because He wanted us to understand a bit of His ways and attributes - on the contrary He very much wants us to exert ourselves on this...
it is interesting to note that scientists are discovering that the entire universe is extremely fine tuned for intelligent life as it exists on earth. Initially they thought the laws of physics and their constants could take a wide range of values and some sort of life would be possible in the universe. Instead they have been surprised to discover that all the physics and laws lie in an extremely narrow range of life permitting values. This persists all the way down to the Big Bang.
there is an interesting book on the subject called "Nature's Destiny" by dr. Michael Denton worth reading if you are scientifically inclined (and know how to put things in the right torah perspective). There he argues that the universe is uniquely fit for life as it appears on earth.
here's the book description on amazon which says it better than me:
While others search the skies for extraterrestrial life, Michael Denton has examined the recent discoveries in all the sciences to ask - Could life elsewhere be substantially different from life on Earth? Drawing on a staggering knowledge of physics, biochemistry, geology, and evolution, Denton builds a step-by-step argument for human inevitability. Life requires water, DNA, and protein; it can only flourish in an Earth-like environment. Building on these claims -which, until recently, were impossible to defend - Denton dares to address the boldest question of all - Is a homo sapien-like creature the only possible highly intelligent being in the universe?
I put a little piece together summarizing some of this here which you may find interesting.
to summarize the world is the way it is because this is the best way for its purpose of being an arena of human free will and teaching man what he is capable of grasping about God.
There are also higher meanings to why things are the way they are.
Just like the torah has multiple layers of meanings so does the world
for example in the Zohar Bereishis:
שלמה המלך, כשירד לעומק האגוז, כמ"ש, אל גינת אגוז ירדתי, לקח קליפת אגוז והסתכל בכל אלו הקליפות שבו, וידע, שכל עיקר התענוגים של אלו הרוחות שבקליפות האגוז, הוא להתדבק בבני אדם
before Shlomo went to take on the forces of evil (Klipot) he looked at a walnut and contemplated its structure of shells which is a sort of map for that underworld.
