I have always found this: "my horse doesn't like men or women or children concept" hard to understand. I am positive that horses make decisions about humans they feel ill at ease with or conversly at ease with, based on factors which we humans do not yet fully understand.
Horses do not see as we humans do, neither do we test a horse to check the quality of its vision.
Horse smell more definitively then we humans do, so they will pick up on smells or odours which we humans have no idea of.
Horses watch and record in memory the repetitive body language which we humans display as we walk or approach a horse.
Horses are very sensitive to touch and just as some humans are wary of getting too close to a horse - so are horse wary of getting too close to strangers.
And then there is voice; the volume and pitch of a human's voice gives clues to horses as to the nature of the individual. Horses react negatively to loud, sharp, fractious, voices.
Then we humans must not try to transfer to a horse our own prejudices about humans which a horse most certainly will not understand.
Any owner of a dog of one of the more protective breeds will tell you that there is a way to approach an animal. A stranger can approach my Rottweiler without hesitation, so long as he first says 'hello' to the dog and allows the dog to sniff him. But there can be neither nervousness nor forwardness in the approach. If we have a deliveryman come to the door and ring the bell then my dog will rush to the hall whilst barking. I'll hold his collar and open the door. Then I'll assess the visitor for myself and mostly I will ask them to say 'Hello' to the dog in an acceptabe manner. Then I let go of the dog who will edge over and will gently say hello to the stranger.
It is the same with horses. There is a way for a strange human to approach and there are ways which will cause anxiety even hostility in a horse.
Yesterday whilst up in the woods a lovely Welsh cob came into view ridden by a young woman followed by a big labrador wearing a dog mask. I called out and said 'hello'. I waited a second or two, I let my dog go and we walked over steadily, dog at heel, to the horse box. I stopped a couple of yards from the horse and then I said 'hello' to it. The woman asked if I would hold the reins whilst she dismounted. The horse stood still, I gave him a stroke and apologised for not having a treat in the pocket of my coat, which probably smelt of mare.
Within a few minutes the five of us, two humans, two dogs and a horse were having a pow wow. We stood, heads to heads, and talked for half an hour whilst the horse was prepared for the journey back home. None of us had met the other before.
The mask fitted to the rider's dog gave a clue that maybe it was aggressive but I could see from its body language that it was not and the rider explained that the dog might otherwise pick up and carry branches which disturbed the horse.
At the time I was wearing a bright blue hand knitted bobble hat, which is very warm and cosy when plonked on my bald head on cold mornings. I must have looked a right twerp. But the horse didn't care. I must have smelt and sounded friendly.
It is rare I hold back from horses, but when I do it is because I can see that my proximity might be unwelcome. In such instances it is usually the rider who is anxious about my interest, not the horse.