I'm just wondering, does this mean he's not fond of any men... or is it something more personal? I've been the one working with my colt primarily, gentling him and training him to lead, pick up his feet, stand tied, yield haunches, etc. There aren't many men around to offer a counterpoint to the female interaction he's had... except the vet who gelded him, which I don't imagine was the most positive experience. He also stood much more willingly for a female farrier to work on him, whereas he wasn't so keen on giving a male farrier his feet. I thought it was more an issue of their training approaches than gender.
Finally, what do I do? My colt is an angel for me to catch. He'll even come to meet me at the gate. Unless I'm able to come out to get him back in though, he won't be able to be turned out.