Let's take a step back: the Hebrew calendar uses lunar months of either 29 or 30 days (for 354 days altogether). Now the Bible says that Passover should be in the spring, and if you keep having years of 354 days you'll keep sliding backwards until Passover won't be in the spring anymore, so every so often they'd add a leap month. Sure, other peoples may have had lunar calendars too, but nothing pagan or Babylonian per se here.
In the books of Moses, the months are simply called "first month", "second month", and so on. Passover is in the first month, Yom Kippur in the seventh. Nothing pagan or Babylonian here yet either.
Throughout the period of the Judges and First Temple (let's say very roughly from 3500 to 2500 years ago), we occasionally see the Bible use Hebrew names for some of the months (such as "Ziv"), instead of just the numbers.
The names that most of us are familiar with are, in fact, the Babylonian names. When the Jews were exiled to Babylon (roughly 2500 years ago) and came back (roughly 2400 years ago), the Jews chose to stick with Babylonian names for the months of the same calendar the Jews had been using for centuries.
So we didn't pick a Babylonian calendar; but we adopted Babylonian nicknames for the months.
The simplest explanation for that was to specifically commemorate the exodus from Babylon, to remind the people that we weren't always in Israel. Exile could happen, but so could redemption. (The book of Chronicles, for instance, ends on the high note of redemption from Babylon.)