Based on the statistics I've seen, if you fall on your noggin, a helmet reduces the chance of serious injury by a factor of 2-5 times, depending on the study. I try to make choices that will reduce the chance of my falling head first, and I do sometimes ride without a helmet, but it doesn't bother me to ride with a helmet and I think it is worth it. It is rocky where I live, and improving my odds 2-5 times sounds pretty good to me. I insist on my daughter using a helmet, but don't lecture anyone else.
However, if anyone tells you helmets are MORE dangerous than going without, they are wrong.
Here are some studies: AANS - Sports-Related Head Injury Spinal injuries resulting from horse riding accidents American Medical Equestrian Associaton