My gelding decidedly does not like most men. He will tolerate certain men, like the farrier (as long as he approaches from the side) and one of the other boarders, and he does love our BO (who basically raised him from a weanling til I bought him, so they do have a special bond). However, if a man walks past him while he's tied, he will pivot to keep the man in his line of sight. If a man comes up to his stall, he will (generally) observe them from the other side of the stall and make no move to be friendly. Even enticements of food (for a growing 2.5yo draft cross) aren't enough to get him to accept most men. It has been tried. Women and children, on the other hand, can do just about anything to him and he could care less. He has never had a bad experience with men to my knowledge. I was told (by the BO, the BO's two sons and one of the guys who kind of works at the barn) that about six months before I bought him, he decided that he just didn't like men. If a random woman were to stand at one end of his stall, a random man at the other, and a random child in the middle, Aires would more than likely go to the child.
I can't say about in the saddle, as the only three people who have ever ridden Aires have been women (myself, the trainer at the barn and my friend).