I think every adult has the right to choose for themselves if they want a helmet or not.
Personally, I always ride in a helmet. I have had plenty of scary experiences myself, but if that wasn't enough convincing, I once witnessed a woman who broke her helmet cleanly in half after her horse tripped at a walk and sent her flying into the arena wall head-first. The horse was as bomb-proof as they come and hadn't spooked or anything else, he just took a bad step and went down on his knees.
Noddy, you talked about having problems with your head getting hot. Have you every tried wetting your helmet/hair? Out on trail during the summer, it's not unusual to see me sponging Dream and then sponging my head or dunking my head outright into the water. Wet hair, while dribbley, is very cooling!