Judaism has special laws that apply only to the Land of Israel.
Wikipedia has a useful summary.
Note for example the
“Laws concerning the products of the land: the heave-offering for the
priests; the tithes to the Levites; the poor man's right to the
gleanings, the forgotten sheaf, and the unreaped grain in the corners
of the field; the use of young trees (prohibited during the first
three years) [also applies outside the Land]; the mixing of
different kinds of vegetables (kil'ayim); the Sabbatical year.”
The Prophet Zechariah refers to the Land as admas ha-kodesh, holy land (2:16)
And the Lord shall inherit Judah as His share on the Holy Land, and He
shall again choose Jerusalem.
The laws applicable to the geographical land seem to me to preclude some other mystical non-geographic meaning.
You may like also to consider that on two occasions, spies were sent to check out the Land. One set by Moshe see Bamidbor 13 (17-20) and later by Yehoshua 2 (1). They were sent to spy out a geographical land.
You ask whether everyone in Judaism agrees to this.
I cannot speak for everyone. The geographic view is surely widely held.