The following is my own "theory" based loosely on, and mostly inspired by, some comments of the Vilna Gaon.
Haman (and Achashverosh) did not know Esther's nationality, but did know that she was raised by Mordechai (the Vilna Gaon makes this point in several places throughout the Megillah. Commentary to 6:1 is one example). Therefore, there was reason for Haman to assume that if he directly assaulted Mordechai, this would raise the ire of Esther and, in turn, the king. Esther had already been made queen before Haman was raised to his position (Esther 3:1), so by the time Mordechai was refusing to bow down to him, Esther was already in her position of influence. Haman would have killed Mordechai immediately, sans any lots, but he was afraid of Esther's reaction.
When Esther invited Haman to the party, Haman concluded that Esther felt more fondly towards him than she did towards Mordechai (Vilna Gaon commentary to 5:9). When, upon his departure from the party he saw Mordechai and was reminded of his hatred for him, he no longer had any barriers preventing him from directly taking action against Mordechai, and went to arrange for his immediate hanging.
I realize this does not fit well with the Vilna Gaon's commentary to Esther 3:6 (which is why I called it my theory inspired by the Vilna Gaon, and not what I think he would say), but, borrowing the explanation of the Malbim to Esther 3:4-6, it could be understood as follows: Haman understood Mordechai's failure to bow as an expression of Mordechai's personal hatred towards Haman (Malbim to 3:5), and even so he decided to destroy the Jews as a whole out of spite for a belief system which could have led to not bowing to him (Malbim to 3:6).