What is the mekor for the very well known restriction of physical contact between men and women? Is it a biblical or rabbinical prohibition?
Different crowds use "I'm shomer negiah" to mean two very different things:
A.) I follow the strictest reading of the law and avoid any touching whatsoever.
B.) I keep the law and don't hug/kiss my unwed sweetheart, unlike many of my friends for whom that's too hard.
Let's address the sources for both.
The Torah starts off the list of prohibited relations with the language lo tikrevu leGalot erva (Leviticus 18:6), "do not come close to uncover shame.". 18:19 applies this to a woman who is ritually impure as she hasn't had a ritual bath after her menstruation; and 18:30 wraps up the whole subject with "don't do any of this messy stuff that the Canaanites did."
Note that virtually all single Jewish women today haven't done the ritual-bath thing, and therefore the biblical prohibition from 18:19 would apply to having relations with them.
Okay, so actual "relations" are out. What about other activities?
Sifra, a Talmudic-era work on Leviticus, reads (13.2): "don't do stuff that gets you close to uncovering shame", i.e. don't engage in behavior that would lead to relations. (Literally: "this teaches that not only is uncovering prohibited, but even coming close to that.") RAM-bam reads this as a biblical prohibition, it's don't-do-commandment #353. Ram-BAN in his commentary on RAM-bam's mitzva-list disagrees; he feels it's a rabbinic prohibition and the Sifra was offering a creative reading to help justify the rabbinic prohibition.
As both Rambam and the Shulchan Aruch codify Rambam's opinion, that generally appears to be the accepted one, that it's a biblical prohibition.
כל הבא על ערווה מן העריות דרך אברים, או שחיבק ונישק דרך תאווה ונהנה בקירוב בשר--הרי זה לוקה מן התורה: שנאמר "לבלתי עשות מחוקות התועבות" (ויקרא יח,ל); ונאמר "לא תקרבו לגלות ערווה" (ויקרא יח,ו), כלומר לא תקרבו לדברים המביאין לידי גילוי ערווה.
One who has relations with a prohibited person, but does so "via body parts" [i.e. not the two types of penetration described in Leviticus]; or who hugged and kissed in a lustful manner and got pleasure from the flesh contact -- such a person can be whipped by biblical law, as it says, "not to do any of these gross actions", and "don't come close to revealing nakedness", i.e. don't approach a person for actions that would bring to revealing nakedness.
Rabbeinu Yonah, however, read the Sifra as prohibiting all touching, not merely the dangerous sort. Many follow this opinion; many try to avoid slippery slopes; and there is much discussion as to what exactly what sort of touching is considered problematic. I strongly recommend Rabbi Yehuda Herzl Henkin's essay on handshakes for more.
PG-13 touching between boyfriend and girlfriend -- prohibited either biblically (Rambam) or rabbinically (Ramban).
Business handshake between two busy professionals who don't know each other from anything and won't see each other again -- prohibited according to Rabbeinu Yonah; permitted according to the way most read the Rambam.
And then many, many shades of grey in between those two extremes.
See Rambam Issurei Biah Chapter 21 and Shulchan Aruch Even HaEzer Chapter 20 who codify this as a Biblical prohibition.
Rambam mentions Leviticus 18:6 and 18:30 as his sources.
Rambam counts it as prohibition #353. Sefer HaChinuch counts it as #188. Semag counts it under prohibition #126.