Divorcing one's first wife is viewed as a particularly sad event. Gittin 90b says that "even the Mizbeach sheds tears" when that happens. The rest of the Gemara there is probably what you heard referenced. The Shulchan Arukh writes (EH 119:3):
לֹא יְגָרֵשׁ אָדָם אִשְׁתּוֹ רִאשׁוֹנָה, אֶלָּא אִם כֵּן מָצָא בָּהּ עֶרְוַת דָּבָר. וְאֵין רָאוּי לוֹ לְמַהֵר לְשַׁלֵּחַ אִשְׁתּוֹ רִאשׁוֹנָה; אֲבָל שְׁנִיָּה, אִם שְׂנָאָהּ, יְשַׁלְּחֶנָּה:
He shouldn't divorce his first wife unless he finds infidelity. And it's not correct to hurry to send away his first wife; but a second wife, if he hates her, he should send her off.
But the Rama there notes:
וְדַוְקָא בִּימֵיהֶם שֶׁהָיוּ מְגָרְשִׁין בְּעַל כָּרְחָהּ, אֲבָל אִם מְגָרְשָׁהּ מִדַּעְתָּהּ מֻתָּר.
That's only in their times when they could divorce [a woman] against her wishes, but if she agrees to it then it's permitted.
In other words, to throw her out on the street against her will requires a very serious breach of contract. However, regarding most cases nowadays, we understand marriages sometimes don't work out and then divorce is the correct option for both parties. Separating from one's first love can be very sad and still be the appropriate action under the circumstances. Anyone having marital problems should speak with trusted counselors, rabbis, etc. to work out the best option for them in their particular circumstances. Don't just run into a divorce, but realize the option is there.