1 - I don't wear helmets that cost more than my horse.
2 - Risk assessment. Any fall, even from the helmet rolling off a chair, COULD damage it in a way that would compromise its future function. Mine has fallen off of chairs more than once, and I've caught grandkids dragging it around. Oh well. I'm not willing to replace it for that. But if I did a lot of jumping, I might replace it for that.
3 - OTOH, if I ever take a fall that is hard enough to break some ribs, I'll probably replace it. Just because. My worst fall came when I was wearing a baseball cap. Since my ball cap had dirt on it, it must have at least touched the ground. Since I didn't feel anything, and since I doubt any of my unorthodox opinions can be blamed on brain damage, my worst fall would NOT have harmed my helmet...but so what? I don't fall often enough to make it an unbearable expense. Which brings up
4 - Any fall has the potential to kill or cripple you. Being 54, maybe I'm just more aware of my mortality. Three falls a year would be enough for me to assess my risk. WHY did I have 3 falls? Am I being careless? Am I overhorsed? Am I trying to do things I lack the training for, or that my horse isn't properly trained for? Or have I accepted the risk for some reason?
I've had two falls in 4 years, both from the same mare bolting during a dismount. I decided that meant my horse lacked training, and we have spent the last 7 months working on the issue of her fear and bolting. She is not perfect, but she is vastly better behaved now, and at least an order of magnitude calmer. That doesn't mean I'm SAFE, but it does mean I'm at reduced risk. We still have more progress to make before I'd try riding her out by herself.
Wearing a helmet is one way of reducing risk. I don't care if people ride with or without a helmet, unless it is on my horse or property - but I do think people should remember that helmets are only ONE way of reducing risk. There are others, and it is reasonable to stop every once in a while and consider them.