Dude I was in the same boat as you 2 months ago. Never rode before always wanted to when I was a kid so I finally did it. Didnt think twice about what people thought. Funny thing is the posts above are correct. GIRLS LOVE IT. My female friend who rides says girls go crazy when they see a good looking dude riding. AND I've had first hand experience at my barn. EVEN WITH A HELMET ON!! Haha. I also first volunteered at a therapeutic riding center...at the orentation ALL chicks I was the only dude...and the one girl even asked my why am I here I was like to help kids ride? And she was like noo whats your angle....i laughed and said there wasent any seriously. They thought I was trying to pick up girl at this place....but man just don't worry what people think and go have fun...its amazing once an horse starts to trust you....hope this helps!!
From another DUDE rider!