Most of the girls/women who take lessons or board at a barn would LOVE to have more young men around. I'm pretty sure you will be accepted with open arms (pun intended).
Once you start taking lessons, you will very quickly learn that this is not a feminine sport. You do not merely sit atop the horse on a little throne and let the horse do the work. It is very physical. You use all parts of your body. You ride with your legs, your seat, your shoulders, arms, and definitely your core. After an hour lesson riding in an English saddle with no stirrups, spending much of the lesson in 2-point, I can guarantee you that you will not think of it as a feminine sport anymore.
It is also a very mental sport. You have to be determined, never show your fear to the horse, and show the horse ALWAYS that you are the leader and will not be messed with. It takes communication, understanding, ability to anticipate, and the ability to stay level-headed when the @#^% hits the fan.
And if the other guys give you flack, I can assure you that playing basketball is neither as mental nor physical as equitation.
You do whatever you want. It sounds like you like some speed and excitement. In the English disciplines, have you looked into eventing or hunters? Eventing is so much fun and so exciting, and requires immense skill and atheticism from horse and rider. Here's a quite snippet of the cross-country stage.
Or do you want to compete at all? Maybe you just will like riding for the fun of it. In Western, you have tons of fast-pace disciplines to choose from, including gaming.
The people and culture if you do, say, eventing vs Western, will be very different. Think about that too. The ambiance at a rodeo is quite different than at a horse-trial. And men participate and win in all of the disciplines. So do women. Welcome to the only Olympic sport where men and women compete together and equally.