I always wear a helmet, whether I'm walking around the barn on a 25 year old retired schoolmaster or a 4 year old green horse. I don't care if it's the safest horse in the world, a horse is still a horse and even if they've been predictable their entire lifetime, it just takes one thing to make a situation dangerous. That being said, I respect other's opinions--no I don't think it's wise at all not to wear a helmet, but that is your decision because it's your life and therefore your choice, not mine.
I've fallen off once, a couple years ago, in 11+ years of riding. I was riding an older (17 or so?) OTTB mare who always leaned towards the slow side, and I was used to her and had been exercising her consistently for a while. Well, for reasons unbeknownst to everyone, she took over at a crazy gallop. Nothing spooked her; we were just having a nice canter like usual and she decided to take off. It was a haze for me, but I remember trying to gain some control and starting to try and take her in circles to control it, but she would have none of that and twisted and threw me in the opposite direction. I was riding in the outdoor so I fell off a 17 hand mare on what was pretty much finely crushed gravel... it hurt! My helmet showed no signs of damage, but sometimes those things are internal so I replaced it immediately. I didn't have any head damage or any soreness or anything, and it pretty much felt like my head brushed the dirt and came to a rest (I fell on my side), but it could have given me a bad concussion if I had hit it w/out anything between me and the ground :)
Now what we need is hip protectors. Hips are usually an old people problem, but even us younguns riding horses can damage them ;) I couldn't put any weight on my right leg/hip for about a week, and it took about 3 weeks till it was just about back to normal xD