Are there any sources that suggest that the incense used in the Bet HaMikdash were hallucinogens?
If this is true, how do we understand their practices back then?
Yosef Glassman offers an interpertation that the "kana bossem" (קנה בושם) mentioned in the book of numbers and in the song of songs as the good kane from the far country is marijuana you can read more about ot here and here
The word "shechar" can mean non-wine alcoholic beverage, or it can mean intoxication. Rashi says that, in this verse, "shechar" means intoxication.
Since wine was often imbibed in a diluted state as a common beverage, Rashi is saying that a sip of diluted wine doesn't prohibit a kohen from entering the mikdash, but drinking wine to the point of intoxication does.
According to this verse, as understood by Rashi, kohanim are forbidden to be inebriated in any way when performing their Holy Temple duties.
Therefore, it would be forbidden to burn incense that would cause any type of inebriation.
EDIT: DoubleAA points out that the topic of whether kohanim may enter the Temple while inebriated is displayed in the Babylonian Talmud, Nazir4a, as a dispute between Rav Yehudah and Rav Shimon. The issue is not resolved there, although the halacha typically sides with Rav Yehudah over Rav Shimon.
That is to say, anything which would impair a kohen's ability to focus on his Holy work, is forbidden to consume before (and certainly during!) the Temple service.