The Rambam writes (Hilchos Isurei Biah 21:1):

כל הבא על ערווה מן העריות דרך אברים, או שחיבק ונישק דרך תאווה ונהנה בקירוב בשר--הרי זה לוקה מן התורה

One who comes upon one of the Arayos via their limbs, or hugs or kisses in a lustful manner and benefits from the touching of skin receives malkus according to the Torah.

The implication is that the category of Shomer Negiah is only applicable with regards to lustful touching. Now I am aware that there is a discussion of cases where lustful pleasure is highly unlikely, e.g. shaking hands in a business setting. However, it seems to be the common practice that men and women will casually touch members of the same sex, since it is assumed that they cannot be attracted to each other at all. What about the case where one or both of the parties has a homosexual, bisexual, or asexual orientation? How will this change the categories of people that may be touched?