I have a gelding who prefers women. I'm pretty sure it is based on his experiences at a ranch in Colorado, since he avoids anyone with a cowboy hat. Over the years, he has learned to tolerate cowboy hats, but a woman without a cowboy hat can approach him much easier. The scars on his side where he was spurred bloody make it easy to understand his feelings.
My BLM mustang shows no preference.
For riding, skill makes far more of a difference than sex. Any of them strongly prefer a good rider to a newbie, although the geldings will tolerate newbies, and the mare has put up with a couple of first time riding guys (but not girls...she despises teen girls). But in the saddle, they care about balance & confidence, not sex.