May one swim on Shiva Asar B"Tamuz? If not, why not? (sources)
The Debriciner, Rav Moshe Stern in Shaalos U'Teshuvos Be’er Moshe 3:77 says it is permissible. The reasoning is that the Shulchan Aruch Orach Chaim 550:2 says that bathing is permitted on the three fasts (Tzom Gedalia, Asara B’Teves, and Shiva Asar B’Tammuz). Although the Acharonim argue on the Michaber and our minhag is to be Machmir, the Pri Megadim says that we are only Machmir for hot water but not cold water.
There are those who argue that swimming should be prohibited since it is (Rechitzas Taanug) bathing for pleasure, which from the Gemara Taanis 13a it seems to be prohibited. However the Debriciner Rov says that Rechitzas Taanug is only prohibited in hot water. If the water is cold, yet you manage to find a way to enjoy yourself, that is not called Taanug.
The Tshuva ends by saying that maybe mourning the destruction of Hashem’s house merits being extra stringent even when the Halacha does not prohibit it.
There are two reasons for not swimming, one applying more than the other:
The Mishna Berurah 549:1 has a piece explaining that the purpose of the fast is a means toward introspection and teshuva. He concludes that those who fast and go on pleasure walks have grabbed the unimportant (tafel) and abandoned the important (ikar).
A second point is brought in 550:6. Although when klal Yisrael accepted the fast days, they did not include washing (rechitza), et. al., that was because most would not be able to handle the extra stringency. However a "baal nefesh" should be strict.
In addition to the other excellent answers, I want to point out one more source because of its direct relevance to the specific question asked.
The Talmud records (Megillah 5b) that Rabbi Yehuda HaNasi publicly bathed in the city of Tzippori on the 17th of Tammuz. Tosfot and Rashba there already point out that he was making a public statement that the strictures of Tisha b'Av (such as not bathing) do not apply nowadays to the other three Temple-related fasts.