From the Gemara, one would learn the relevant Rishonim, Achronim, Teshuvah literature, and later Halakhic works... there's quite a lot to go through on any given 'sugyah' (Talmudic topic). 

Regarding the choice that a posek is faced with when there are differing opinions on the matter, there are a few general rules aptly referred to as 'kelalei hapsak' (general rules of paskening) which are scattered throughout the Gemara and Rishonim (like any halakha) and have been collected by certain Achronim, such as by the Peri Megadim and the Maharit Algazi, and more recently, Rav Ovadia and Rav Yitzchak Yosef (in the introductions to Yalkut Yosef, and in a major three volume work called 'Ein Yitzchak'). 

These rules are varied and detailed, but some major ones are: following the majority (not necessary a quantitative majority; some 'greater' earlier poskim might outweigh numerous others), following the Shulchan Aruch and Rama, and following well established customs. Different poskim will give different weights to these considerations, for example, Rav Ovadia Yosef gives much stronger weight to the Shulchan Aruch. Of course, there's also the question of what makes the most sense. Rav Moshe Feinstein was wont to argue on earlier poskim becuase he thought he had a better explanation of a Gemara than they did, for example. These 'shikulim' (measuring/weighing) aren't clear cut and different people feel differently about these rules, which is why we follow poskim (as opposed to making our own decisions) in the first place; only someone with the appropriate breadth and depth of knowledge can be assumed to have the proper instincts about when and how to apply these rules.