Why was God's injunction through Zechariah, on rejoicing, not followed?

Question

With the return home from Babylon, the Temple rebuilt, and idolatry vanquished, God said, through the prophet Zechariah, that from now on the four fast days commemorating the destruction of the First Temple (10 Tevet, 9 Av, Gedalia, 17 Tammuz) should become joyful days:

Thus said the Lord of Hosts: The [four fast days] shall become to the house of Judah days of joy and gladness, and happy feasts. [Zech. 8:19]

This is abundantly clear, but the Talmud interpreted this injunction as follows:

When there is peace, [these days] shall be days of joy and gladness. When there is persecution, they shall be fast days. When there is no persecution but not yet peace, [these fasts are optional]. [They asked:] If that is the case, [should the fast of] Tish'a b'Av also [be optional]? Rav Papa replied: [No.] Tish'a b'Av is in a different category, because several misfortunes happened on it... [Rosh HaShana 18b]

But today these four days are still sad and mandatory fast days, even in times of "peace" and "no persecution". How can the Gemara essentially nullify such a clear prophetic directive?

Note added: There is at least one opinion that the Gemara indeed changed things, because the people stopped fasting "in the Second Temple era":

When the exiles returned from Babylonia to build the Second Temple, these fasts were canceled and transformed into holidays. [Peninei Halakhah, Zemanim 7:1:3]

Indeed, during the Second Temple era, these days became joyous festivals. [Peninei Halakhah, Zemanim 6:1:4]

Answer 1

There seem to be two questions here: how we get that there's a difference between when there's peace and when there's not, and why in practice we fast even in times of peace.

For the first, the answer is in the couple of lines of the Gemara preceding what you quoted:

מַאי דִּכְתִיב כֹּה אָמַר ה׳ צְבָאוֹת צוֹם הָרְבִיעִי וְצוֹם הַחֲמִישִׁי וְצוֹם הַשְּׁבִיעִי וְצוֹם הָעֲשִׂירִי יִהְיֶה לְבֵית יְהוּדָה לְשָׂשׂוֹן וּלְשִׂמְחָה קָרֵי לְהוּ צוֹם וְקָרֵי לְהוּ שָׂשׂוֹן וְשִׂמְחָה בִּזְמַן שֶׁיֵּשׁ שָׁלוֹם יִהְיוּ לְשָׂשׂוֹן וּלְשִׂמְחָה אֵין שָׁלוֹם צוֹם

What is the meaning of that which is written: “Thus said the Lord of hosts: The fast of the fourth month, and the fast of the fifth, and fast of the seventh, and the fast of the tenth, shall become times of joy and gladness, and cheerful seasons, to the house of Judah” (Zechariah 8:19). It calls them days of “fast” and it calls them “times of joy and gladness.” How so? When there is peace in the world, they will be times of joy and gladness, on which eulogies and fasting are forbidden; but when there is no peace, they are days of fasting.

Rashi there defines "peace" as "when the non-Jews aren't in power over the Jews." Rabbeinu Chananel defines it as when the Temple is standing. Neither of those obtains nowadays. (Note that, by contrast, Zechariah's prophecy was while the Second Temple was being rebuilt - it was completed two years later - and later during the Second Temple era there were periods when the Jews had full independence.)

As for the second question, which is where Rav Papa's three-way distinction (peace, persecution, or neither) comes in - for that we have Magen Avraham, Orach Chaim 550:1, who says:

אע"ג דמדינא בזמן שאין סכנה אין חייבי' להתענות אלא בט"ב מ"מ כיון שנהגו להתענות כולם אסור לפרוץ גדר

Even though legally, when there's no danger, we're not required to fast except on Tisha B'Av, nonetheless since everyone has made it customary to fast, one may not "breach the fence."