No it's just a horse thing. You shouldn't make excuses for your horse. Some horse like people some could care less. The more you do with your horse the less standoffish he will be. Don't blame the gender.
Also - how do you approach him? He's a mini - you are a human. I will guess you 'tower' over him. With some horses the flight instinct is very strong. Try inviting him into your space.