Yes. It is a dangerous sport. It's unpredictable.
Everything can be 'dangerous' if you think about it. Getting into a car. Walking into a store. Etc. Life is unpredictable, really.
But I honestly don't even think about it when I ride. If I get injured, something happens unexpectedly, at least I was happy.
After all, you can't get too anxious/nervous when riding because horses can totally sense the energy you give off. Have to stay calm and relaxed.
Whether you wear a helmet/protective gear or not, that is up to you. I always wear a helmet.
I would not worry too much about what 'could' happen. Just ride, and have fun! Take it day by day, and don't do anything you're not comfortable with. No pressure to do anything insane. You don't have to do jumping, eventing, etc. you can ride for pleasure- but that doesn't exactly mean it's 'safer'. Anything can happen, like I said. Unpredictable.
What's kinda silly, but true is that most of my falls (I haven't had too many lately, but I have before) is when I'm at a walking pace!
For example, went to open a gate one time on horseback and I lost my balance & fell right off. I was fine, but just a bit bruised up! LOL.
I have fallen while jumping, but not too often.
Bottom line: Just enjoy it. Even when you AREN'T in the saddle, grooming/bonding with a horse is just as fun. :) At least I think so.