As the berachah on the havdalah candle is made, I've been told, the proper thing to do is to raise your hands toward the candle with your fingers curled over your palm and look at the shadow of your fingers, which are cast onto the palm. Where is the halachic source for this?
First of all, the specific language of what you've heard, that "as the berachah on the havdalah candle is made..." is not necessarily correct; either the bracha should be before the checking, or the fingernail-checking should be done before the blessing, but not at the same time. (As to which one comes first: R. Moshe Feinstein, in Igros Moshe O.C. 5:9 writes that the correct custom is to make the blessing first, and only then look at your fingernails, but the Mishna Berura says that it is more correct to do the opposite, to look, then bless).
The Mishnah in Berachos 51b writes the blessing of "Bore Moerei Ha'esh" should only be made if one benefits from the light of the candle. The Gemara later on daf 53b writes that (according to the accepted opinion) this means that one has to get enough use out of the light as to differentiate between two coins: a sela and a pundyon (these two coins are actually very similar looking to each other).
Since we don't usually have coins on us at the end of Shabbos, the Tur (O.C. 298) writes that instead one should use the light to differentiate between their fingers and fingernails. (The Gr"a to Shulchan Aruch provides an earlier midrash as the source). Besides for this halakhic reason, the Tur gives other reasons as well: nails are a symbol for growth and proliferation, because the fingernails grow. (Many quote other mystical reasons as well).
There are some interesting aggadic sources to specifically look at your nails:
1) We are asking for the week to be fruitful and multiply- like nails, that continually grow (mentioned above).
2) That the fire created after shabbos came out of the fingernails of Adam. (Kol Bo 41)
3) In בפרקי דר״א פי״ד, it says that Adam's clothes were made of nail like materials. After the sin, his clothing fell away and he realized he was naked. This explains Hashem asking Adam "who told you that you were naked ? did you eat from that tree ?" Showing the connection.