In Bamidbar 25:16-18, Hashem commands Moshe to take revenge on Midian for having caused the Jews to sin at Shittim, discussed at the beginning of the chapter. Yet v. 1 seems to indicate that the Jews sinned with Moav as well:
וישב ישראל בשטים ויחל העם לזנות אל בנות מואב
And Yisrael dwelled in Shittim, and the people debased themselves to commit harlotry with the daughters of Moav.
So, if Moav was involved with the incident at Shittim as well, why was the command to attack directed only at Midian?
The Kli Yakar (to v. 17) translates ויחל not as "they debased," as I did above, but "they began." Thus, the war was not with Moav, because the Jews started up with Moav and not the other way around. Midian, on the other hand, started up with the Jews, and therefore the Jews were able to retaliate.
He cites and explains a Sifrei (Ki Seitzei 117), also in line with translating it as "they began," insinuating that "they began" with Moav but concluded with Amon. The same way that Lot's elder daughter was more explicit in her illicit relations by naming her son Moav, "from father," while his younger daughter hid the act behind the name Ben-Ami, "son of my nation," so, too, Moav's involvement in Shittim was explicit but Amon's was hidden. We see this in Devarim 23:4-5, where the prohibition to marry Moavi and Amoni converts is explained as being because they did not give you bread and water and because they hired Bilam. Why didn't they give bread and water? In order to lure them to the shops set up in Shittim for food, which lead to the sin.
The question remains: Moav did start up with the Bnei Yisrael - they hired Bilam to curse the Jews, and the whole Shittim incident was Bilam's idea when cursing the Jews failed (Rashi to v.1). Further, based on the Sifrei he quotes, Moav withheld food from the Jews in order that they sin at Shittim. So how can he say that Bnei Yisrael started up with Moav, when in fact Moav started up with Bnei Yisrael?