Rabbi Prof. Yoel Elitzur wrote in his book "Makom Ba'Parasha", pg. 425 (my translation):
"First of all, we must remember that Yerushalayim was only chosen in the time of David and Shlomo. Prior to that, the Tabernacle was in different places, but the mitzva of the work of Yom Kippur always existed. Meaning, "Azazel in the desert", if that's a title with geographic significance (and not the name of a demon or angel, as many of the ancients and moderns thought), then it's not a specific location in the world but a general title for a place in the desert that has a cliff from which the goat can be thrown, as it says "to an inaccessible region; and the goat shall be set free in the wilderness" (Vayikra 16:22). And from here we see that the cliff that the goat was led to near Yerushalayim was not [commanded] from the Torah [me'de'oraita]."