Is a man allowed to talk to a woman for non shidduch/business purposes? Which halacha could I look up to get more info on this?
What an important and difficult topic!
First Look at Kiddushin 70a-b and the commentators about the prohibition of inquiring about a woman's situation (sh'eilat shalom--a topic which includes speaking to them), particularly the language of Meiri and an important Ben Yehoyada; and see Bava Metzia 87a in the middle of the page. See Shulchan Aruch 21:6 and Aruch Hashulchan on that chapter; also see the last Ritva in Kiddushin (82a) who is quoted in Aruch Hashulchan. See Rabbi Nissim Karelitz in Chut Shani on Even Haezer Chapter 21:6, Note 16 on pg. 59 and in other places in that work (http://hebrewbooks.org/48274). Also, the beginning of Mishna Avot (Mishna 5) says not to excessively chat with women because it ruins your life in many ways.
An interesting discussion for the Jewish sociologists: http://www.theyeshivaworld.com/coffeeroom/topic/saying-good-shabbos-to-girls-men/page/4.
In a strict halachic sense, the main issue, as Rashi on that gemarah in Kiddushin says about talking to a woman (definitely to who is an erva and maybe even one who isn’t) is the creation of closeness between a man and woman (and is quoted by Aruch Hashulchan). This may possibly be because of the bad thoughts that accompany it and/or the possibility of it leading to worse sins. Nonetheless, having derech eretz is the first priority (as I heard from a Rav in the name of R Yaakov Kamenetsky and as is stated in Aruch Hashulchan and in Ben Yehoyada), so saying 'hi' or 'how are you' in America (or the equivalent greetings in other cultures) is usually allowed. The application of how you should converse will differ depending on the social circumstances: Many Hassidic and Yeshivish people don’t talk to the other gender at all (even the work place has very limited talk), some religious circles don’t refer to women by their first name, other religious Jews are more lenient for derech eretz reasons to say hi or shalom but try to avoid excessive closeness (Chut Shani is strict with saying shalom because it probably creates closeness), other religious Jews have friends from the opposite gender (which seems to be halachicly incorrect, unless there are unique practical reasons for the friendship other than desire); Western secular societies have no such boundaries.
Based on Rashi's definition (and reasons of maintaining peace and derech eretz, making a living, and accomplishing essential life needs), it seems that practical relationships like mitzvah, business, shidduch related, and familial relationships are permitted, yet should be kept under control (i.e., avoid excessive chatter and levity), and should not go against the accepted custom of the community you live in.
There is a defense for Jews who are lenient with having platonic relationships with non-married (or non-erva) women, based on what R Moshe Feinstein (Book 9: Even Haezer 5 Letter 16) told his son R Reuven--that the prohibition in the Talmud (of sh'eilat shalom) may only be by married women and not by non-married women (see Meiri ibid.). Practically he rules to be strict in general, especially considering that most young women are a Niddah. He also rules in another letter (Even Haezer 4 Letter 60), that it is forbidden to have platonic relationships because of the sins it leads to, and does not believe that “halachic” platonic relationships really work (http://www.scribd.com/doc/22288885/R-Moshe-Feinstein-on-Platonic-Relationships). I am merely defending the practice (but not endorsing it) based on his idea. Anyway, I don't see the need for having this type of relationship if it is not used to accomplish practical and permissible goals.
Also, if a man has issues dealing with thoughts, I would advise him to avoid talking to women unless needed for derech eretz, business, or other essential needs, and should try to get to the bottom of why he is plagued by thoughts when around women, so he can deal with them in a healthy manner.
The grandson of Aryeh Levine, reports how he witnessed Rav Shlomo Zalman Auerbach zt'l would go to the women's section during simchas to wish them a mazel tov. He says Judaism used to be alot more normal back then.
could be today people have a harder time talking to women without thinking sexual thoughts
Here is some info. You weren't clear if the issue you are concerned with is seeing them or hearing them so here are some random sources.
Reb Moshe in Igros Moshe Orach Chaim ch. 1 siman 40 writes in a tshuva to be careful not to look at a woman's face even when talking to give her a halachic ruling.
There is a long eizer mikudash (the butchatcher rav, printed in the shulchan aruch) in hilchos yichud where he mentions his practice to loudly say mazal tov twice by a shalom zachor, being that one is for the mother but its not proper to talk to her directly, so he would just yell it out twice.
Whether a woman's speaking voice itself is called an erva, see the achronim on the Shulchan Aruch Orach Chaim siman 75 siff 3. They seem to be lenient.