There are lots of differences. Among the more obvious ones:
Dialect - the Bavli is written in Eastern Aramaic, the Yerushalmi in Western Aramaic. There are differences in vocabulary (such as B. חזי = Y. חמי, both meaning "see" - we in fact use both of these in the second Kol Chamira on Erev Pesach morning), in word forms (the Bavli tends to drop final ן in words like תנינא, the Yerushalmi usually preserves it), and in technical terminology (such as B. תא שמע = Y. נישמעיניה מן הדא).
Presentation: the Yerushalmi is a lot briefer, and doesn't have a lot of the extensive discussion and parry-and-thrust found in the Bavli.
Editorial: sometimes the Yerushalmi jumps from one topic to another without explanation (as in Sanhedrin 10:1, where after mentioning the comparison of Torah to nails it asks, without preamble, "How many nails can be in it?" referring to an issue about wearing hobnailed boots on Shabbos), while the Bavli will usually preface the new topic with an appropriate quotation from a Mishnah or the like. Similarly, the Bavli often spells out the corollaries that can be deduced from an Amora's statement, where the Yerushalmi won't do so.
Coverage: the Yerushalmi covers all of Seder Zera'im while the Bavli doesn't. Conversely, the Bavli covers most of Kodshim while the Yerushalmi doesn't. (In both cases, though, this may be due to missing parts.)
Commentaries: the Bavli has Rashi, the Yerushalmi has other commentaries (the standards in most masechtos are Pnei Moshe and Korban Ha'eidah, which certainly explain it adequately - but after all, there was only one Rashi).
The reason they developed separately is simply that in that era there were two major centers of Jewish life and of Torah study, in Eretz Yisrael and in Bavel. They exchanged information constantly (through the נחותי, "traveling scholars," mentioned often in the Gemara, like R' Dimi and Ravin), but still, each of them had its own distinct style of learning (see Bava Metzia 85a and Sanhedrin 24a). The Yerushalmi's development ended much earlier (and some historians say that in fact it was never really properly finished) because of Roman persecution that nearly emptied Eretz Yisrael of its Jews.
There are occasional differences in halachah between the two Talmuds, and when that's so, we generally decide according to the Bavli. The main reason for this is that, as above, the Bavli was finished later than the Yerushalmi, so הלכה כבתראי - we follow the later authorities (since they had access to the earlier material and consciously differed with it).
There are historians who argue that the differences between Sephardim and Ashkenazim reflect the earlier differences between the Jews of Bavel and Eretz Yisrael, respectively. Others argue that it goes back to the differences between the Jews of southern vs. northern Eretz Yisrael. Still others attribute most of them to the differences between the surrounding cultures in which they lived - Christian vs. Muslim. Historically, it is certainly true that the "Romaniote" communities of the Byzantine Empire were more influenced by Eretz Yisrael, and indeed some of them focused more on the Yerushalmi - I believe that the Yanina community (in Jerusalem and in New York) still does so. But then the question would really be whether the Romaniotes were the forerunners of the Ashkenazim.