The difference is discussed extensively in the eighth perek of Ketubot. נכסי מלוג refers to assets that a woman brings into the marriage, or which falls to her as either an inheritance or as a gift after she has married, but which are not included in the value of the ketubah. The word melog comes from the Aramaic מליגה, which means "plucking": her husband has rights to these assets, but may not use them up entirely. He also, therefore, need not replace anything that he uses.
נכסי צאן ברזל, on the other hand, refers to assets that she brings into the marriage by way of a dowry. Their being set in stone, so to speak, refers to the fact that her husband may use as much of them as he wants, but that he will always be liable to replace anything that he uses, and the assets in full will revert into her possession with his death, or with his giving her a divorce. This term is also used in a non-marital context, as for example in the issue of lending money (Bava Metzia 5:6) - there, the mishna notes that one may not receive "צאן ברזל" from a Jew, since the use of his property with responsibility to recompense him for anything used and with zero liability for accidental loss comprises a case of ribbit.