I think it depends on what they get used to. As well as, there has to be trust and/or respect between the horse and the rider.
I have a horse that I have had since he was 22 months old. I trained him myself, I am the only one that has ridden him, I am the only one that takes care of him. I spend hours with him everyday. He is nine years old now. We have been on hundreds of miles of trails, all over the place. He completely trust me and I him.
My husband and I was going on a trail ride with some friends and I was trying to pack up the rest of the stuff in the horse trailer. I asked my husband to brush my horse down to save time. My husband informed me that my gentle horse that I never have trouble with, pinned his ears back, attempting to nip at him , cow kicked at him while he was trying to brush him down. My husband said that he could tell that I was walking up because my horse started watching the barn door and straighten up and acted as if everything was fine as soon as he seen me.
This horse is so spoiled to me that he will not let any of our other horses near me when I walk out in the field with them. Soon as I go to the gate, he is running to the gate and I have to catch him first and put him into the stall so that I can catch the other horses.
So, I am thinking if you want this horse to trust you and get attach to you, then you need to be the one to do everything for this horse. I did a lot of round penning and teaching ground manners and control before I ever ride my horses. I spend time with them every day, brushing while their eating, talking to them, cleaning out their stalls while they are cross tied watching me. They are completely depended on me.
I think that if you spend a lot of time with him, take time to do a lot of ground work and get your confidence while working with him. He will trust you and it will not matter to him if you are a male or female.
Good luck with him, I hope that you will have many happy trails with him.
The true champion in any sport possesses that special potion of courage, determination and pride that cannot be bought or manufactured and is reserved for only the very few.