Are there any sources that suggest that the incense used in the Bet HaMikdash were hallucinogens?
If this is true, how do we understand their practices back then?
Do not drink wine "v'shechar", neither you nor your sons with you, when you go into the Tent of Meeting, so that you shall not die. [This is] an eternal statute for your generations. (Leviticus 10:9)
The word "shechar" can mean non-wine alcoholic beverage, or it can mean intoxication. Rashi says that, in this verse, "shechar" means intoxication.
Since wine was often imbibed in a diluted state as a common beverage, Rashi is saying that a sip of diluted wine doesn't prohibit a kohen from entering the mikdash, but drinking wine to the point of intoxication does.
According to this verse, as understood by Rashi, kohanim are forbidden to be inebriated in any way when performing their Holy Temple duties.
Therefore, it would be forbidden to burn incense that would cause any type of inebriation.
EDIT: DoubleAA points out that the topic of whether kohanim may enter the Temple while inebriated is displayed in the Babylonian Talmud, Nazir4a, as a dispute between Rav Yehudah and Rav Shimon. The issue is not resolved there, although the halacha typically sides with Rav Yehudah over Rav Shimon.
Rav Yehudah said that [a kohen] who eats preserved figs from Keilah, or drinks honey or milk, and then enters the Temple, is guilty (Babylonian Talmud Nazir4a)
That is to say, anything which would impair a kohen's ability to focus on his Holy work, is forbidden to consume before (and certainly during!) the Temple service.