When someone says Hashem's name in error, why do we say Boruch Shem Kevod Malchuso L'Olam Voed - ברוך שם כבוד מלכותו לעולם ועד?
The above line was initially uttered by Ya'akov Avinu on his deathbed as a response to the unified proclamation of Shema Yisrael by his sons (Pesachim 56a). In the Temple, when the kohein gadol would pronounce the shem ham'forash, the people would respond by uttering this line (Yoma 66a).
The Gemara explains this practice as follows (ibid, 37a):
In fact, this, rather than amen, was always the standard response in the Temple because that is the only place where the shem havayah was uttered (Tosefta B'rachos, 6:22; Sifsei Chachamim, D'varim 32:3). According to the Tosefta, the use of this specific formulation is alluded to in Nechemya 9:5, where the Levites said:
Furthermore, the Midrash states that Moshe Rabbeinu ascended to heaven and observed the angels praising HaShem with this specific formulation (D'varim Rabba, 2:36; Tur, Orach Chayyim 619). Moshe therefore recognized that this specific formulation should be used.
The source for reciting this line after a b'rachah l'vatalah is the Yerushalmi B'rachos, 43b:
The Rambam holds that if HaShem is praised as an antiphon to a mistaken utterance of His name, the utterance is not considered l'vatalah. This extends to praising HaShem with any similar phrase, such as "He is Blessed forever" or "He is exceedingly great and laudable" (Mishneh Torah, Hil. Sh'vu'os 12:11).
The Mishnas Ya'akov commentary argues that, in the special case of a b'racha l'vatalah, the Rambam would agree that one must use the specific formula stated in the Yerushalmi. The Tur and the Shulchan Aruch in fact codify the requirement to recite ברוך שם in such a case (Orach Chayyim, 206:6).
It says that in case of fear for blasphemy we have sanctify the Name of G-d by these words. See the whole article for more details how these words helps to sanctify the Name of G-d.