Is one liable to death by the Sanhedrin if they killed someone using the Shem HaShem or if they used kishuf.
--Part 1: Aggadah/Kabbalah
As explained in Who says Moshe sinned for killing the Egyptian and why, the Zohar (Raayah Meheimna, Parshas Mishpatim) explains that one who kills in this way is liable for the death penalty. Here is my rough translation (copied from there), with emphasis on the pertinent part:
With this introduction, these Pesukim can be explained, when they say "He who hits a man and he dies" (מכה איש ומת מות יומת, which is an Aveirah that is punishable by death), this refers to Moshe Rabbeinu, who killed the Mitzri called "ISH Mitzri", and if this would have been purposeful, his judgment should have been to be killed. However, since the purpose of the intention of Moshe was to help and fix his Neshama by killing him with the Shem Hameforash, if so, he killed him Beshogeg (i.e. since his intentions were for good, even though he killed him, it is not punishable by death, rather by Galus), and this is why it continues "Vaasher Lo Tzadah", which refers to Moshe...
Interestingly, there is a Gri"z Al Hatorah that suggests that this action was "Misah Bidei Shamaim", punishing the Mitzri for hitting the Jew. According to this (that it is considered a "heaven"-imposed death), I would imagine that one could argue that he should be exempt.
--Part 2: Halacha
Here is the authoritative Teshuvah on the topic, from Harav Warhaftig. Among the highlights:
The Steipler (Kehillos Ya'akov, Bava Kama 39) concludes that one would be Chayav according to Halacha:
לא מצינו ראיה לפטור את הורג או מזיק לחבירו ע"י שד (נראה דצ"ל – שם) או ע"י כישוף, ומצד הסברא הי' נראה לחייב כל היכא שהוא בכח ראשון, ואף שלא עשה מעשה בשל חבירו רק דיבורא בעלמא, הרי התוס' ב"ק (ק,א) כתבו דכל היכא דבדיבורו קם דינא, חייב כדין אדם המזיק
The Chida and Halachos Ketanos conclude similarly (sources in the article), although Rav Warhaftig argues the possibility of exempting such an individual based on various Halachic arguments. He also makes a similar suggestion to what I mentioned above in the name of the Gri"z.
...And here's another comprehensive discussion, again in Hebrew.