One thing you have to consider when you discuss the people who were exiled after the desctruction of the first temple is that they were exiled to Bavel. Of all the places they could have gone to, they were exiled to a place where they spoke the language and were able to communicate with other people already there. That had a big impact on a lot of people.
Another thing to consider is that although the second temple was built, many rabbis knew that this was also temporary. There is an injunction that if you're about to plant a seedling when someone comes to tell you that the Messiah has come, you should plant the seedling first. Jews have been very wary about following one leader en masse (even Moses only managed to convince a small portion of Jews to leave Egypt with him; the rest died during the plague of darkness), and even more so when they knew the final redemption was not yet here.
I think the main criteria for a lot of major Jewish personalities of the time was that they should settle in Bavel, and still give honor to the temple while it stands. The gemara is replete with stories of rabbis who traveled to or from Israel to relate a teaching from one group of rabbis to another. If being around the temple was the ONLY major reason for them to be in Israel, then it could be asked why so many Jews simply didn't go up.
