There are a lot of good reasons for wearing a helmet. Saying, 'I don't like the way it feels'... is kind of stupid, in my opinion. That's like driving a car and not using the seatbelt because it bothers you. Are you old enough to make that decision? Of course you are. Problem is, it doesn't mean it's the right
I choose not to wear a helmet. I ride and train my own horses, and have an unusual sense of balance. I rode and broke horses right off the race track with a ex-jockey, and after getting thrown multiple times, somehow I learned to stay on for the worse of it.
My horses I have now do not act up. They do not rear, or buck. He can certainly stumble, but I try to keep my horses fresh enough so they never feel tired enough to stumble because of tired muscles. They spook from time to time but they don't really go anywhere, and I've had them for two years. However, saying 'I trust my horses' is a pretty stupid thing to say, too. How many times have you heard a horse owner say 'I NEVER would have thought my horse would do that!!' You hear it DAILY. People do not understand horses as much as they think they do, half of the time. But, I digress. I ride my horses calmly, but at the same time I am /always/ prepared. (It also helps that I mainly ride them in fields... no roads or fences to get thrown off into!)
I am not against wearing helmets, though. If I ride an unbroke colt, or a horse that is known to act up--sure, I'll throw one on. It only makes sense. Sometimes I wear one if I'm jumping, but only if the jumps are bigger then a foot and a half (or if it's someones horse).
Mainly, I don't because I have confidence in me and my horses, but I own four so I always have one on hand if needed.