R. Hershel Schachter has discussed this issue a number of times, and explains it thus:
It is based on technicalities of kinyanim. One of the kinyanim that is used for selling chametz is a kinyan chatzer. A potential issue raised with this kinyan is that a kinyan chatzer works as a shliach, and a non-Jew does not have the ability to create a shliach. However, this is only true for a man. For a woman, a chatzer works as a yad. Therefore it is possible that a kinyan chatzer with a non-Jew only works if the non-Jew is a woman.
He says that because of this R. Moshe Sofer would specifically sell the chametz to a woman. You can hear him explain this, among other places, here starting at approximately 27 minutes in.
(This is an oversimplification, so you should listen to the shiur. Even the shiur is an oversimplification, so perhaps a good place to start is the Ketzos 194:3.)
The original source for this Chumra is the Machatzis Hashekel OC 448, s.v. Vetzarich Iyun (see the full discussion for details):
ולכן עלה ברוחי דעדיף למכור לאשה גויה שאין לה בעל, דכהאי גוונא איכא למימר דמודה תשובת אמונת שמואל דחצרה קונה לה, דמבואר בחו"מ סימן רמ"ג [סעיף כג] דקטן אין לו חצר, דלגבי איש חצר מטעם שליחות אתרבי ואין שליחות לקטן, אבל קטנה יש לה חצר דגבי אשה חצר משום יד אתרבי וקטנה יש לה יד. ואם כן אשה גויה יש לה חצר דהא אית לה יד.