In Parshas Matos (Bamidbar 31:8) it says וְאֵת בִּלְעָם בֶּן-בְּעוֹר הָרְגוּ בֶּחָרֶב.
Did Bilam do anything to avoid getting killed? What did he do? And who actually killed him?
Rashi (to Num. 31:6) quotes a Midrash that Bil'am used magic to lift himself and the Midianite kings into the air, but that Pinchas counteracted that by displaying the tzitz, causing them to all fall down - literally על חלליהם, "on the corpses" (of the other Midianites) as in v. 8.
Radak (to Josh. 13:22) cites this as well, and adds that Pinchas actually killed Bil'am.