The Encyclopedia Talmudit entry on “Bas Qol”, the paragraph about its impact on halachah focuses on the apparent conflict between the conclusion one would get from the Tanur Akhnai story and from the one about following Beis Hillel. A summary of resolutions:
1- Rav Nissim Gaon (Berachos 19b), opinion I: The bas qol said “halachah k’moso b’chol makom”. As a general rule, the halachah is like R’ Eliezer, but not here. The halachic conclusion does not contradict the bas qol, and it’s even possible that the bas qol caused them to reach their decision.
2- Ibid, opinion II: The bas qol was only a test for the sages. Again, normally bas qol would indeed have halachic say.
3- Tosfos (Eiruvin 6b) I: The bas qol was only for the kavod of R’ Eliezer, who called down the opinion of Shamayim. bas qol does NOT have halachic authority.
It seems to me #3 is only possible (assuming that G-d doesn’t lie) by saying that R’ Eliezer and R’ Yehoshua were in an eilu va’eilu situation — both were right. Therefore, to show R’ Eliezer respect, G-d asserts that R Eliezer isn’t wrong even though the halachah is like R’ Yehoshua. In short, exactly the same point made by the Beis Hillel vs Beis Shammai story.
4- Tosfos II: There is a difference between whether the bas qol runs counter to metahalachah (normal halachic process), or in accordance with it. Bas qol can confirm a ruling, but not run counter to normal halachic process.
Metahalachically (i.e. in terms of the halakhos about how to make halakhah), we follow Beis Hillel because they are the majority. The bas qol only confirms that fact.
(Why did it need confirmation? I would conjecture that this is because this is the first generation that the Sanhedrin was in exile from the Temple, and because Beis Shammai were generally considered the sharper group. Therefore there was a crisis in confidence in rejecting Beis Shammai’s opinion without the conclusion coming from the Chamber of Hewn Wood.)
5- Or Samei’ach (Yesodei HaTorah 9:4): There is a distinction between whether the bas qol is clarifying a particular halachah and whether it speaks of a person’s ruling. In the first case, bas qol is certainly not followed — metahalachah is the G-d-given means of creating new halachah. (cf Temurah 16a, where the prophet Yehoshua refuses to retrieve lost halachos via prophecy.) In the second, we do follow Beis Hillel, as per the bas qol. (Although R’ Yehoshua disagreed about this use of bas qol as well.)
No. 5 appears to be nearly identical to #4, but with the added statement that given two true answers (speaking of one of two extant rulings), i.e. metahalachah allows one to follow either, a bas qol can be followed. His conclusion is that even had Beis Hillel and Beis Shammai been of equal number, the halakhah would still be like Beis Hillel.
In short, Rav Nissim Gaon gives authority to bas qol to override halachic process, and the Achnai story’s bas qol is a special case for two different reasons. Tosafos and the Or Sameiach agree that bas qol has less authority than metahalachah, and possibly even no halachic say at all. And it is their opinion that one finds again and again in later sources.
(In general, the geonim had an understanding of halachic truth and of machloqes that none of the rishonim continued with. This may be a consequence. See R' Moshe Halbtertal's "The History of Halakhah, Views from Within: Three Medieval Approaches to Tradition and Controversy")
In either case it’s a question of whether one follows pre-existing rules for making halachic decisions despite supernatural evidence. It’s support for the notion of meta-halakhah, not for arbitrary leeway in making decisions.
