Suppose I am having a conversation with a person who is unfamiliar with Jewish tradition, and the person casually attempts to pronounce the Shem Hameforash according to its letters, as some secular historians do. Should I utter a response of some kind, such as ברוך הוא וברוך שמו οr ברוך שם כבוד? Does it matter what the attempt sounded like?
It depends on if the person believes in God in the way Jews do: As a single entity with no shituf (so this excludes most, but not all, Christians).
However, should a Jewish heretic write a Torah scroll, it and the name of God it contains must be burnt, since he does not believe in the sanctity of [God's] name and did not compose it for this purpose. Rather, he considers this to be similar to any other text. Since this is his intent, the names [of God he writes] do not become holy.
In contrast, if a gentile writes [God's] name, it should be buried.
So it depends on the intent of the speaker when he said the name.
... answering “Amen” to the bracha of a kofer/apikoris, an individual who denies a fundamental yesod of Judaism, such as Torah from Sinai, or possesses a distorted idea of God. As noted above, the Rambam clearly indicates one should not answer “Amen” to such a bracha, and based on the above explanation, it would seem this individual lacks the ability to create any true halachic statement as it relates to God.
You have a separate problem that the person is unlikely to have pronounced the name correctly. So it is probably not God's name anyway.