With horses, it's not 'if' but 'when'. Truthfully, if you're around them long enough, you're probably or even certainly going to get hurt at some point.
It's a risk you have to decide to take. Minimize your risks by wearing body protection, and being as careful as you can, but don't stop enjoying your horse just because you're worried you might
It seems sometimes like it happens when you least expect it. I've fully expected to be hurt a few times on younger, more reactive horses, but ironically, it's my old bombproof kid's horse that I've fallen from more times than I can count (doing stupid stuff that I probably shouldn't have been doing, of course
) and knocked myself silly.
I wish I had some motivational speech to share, but all I can say is confront your nerves and don't let them get the best of you. Don't take dumb risks, and prepare yourself for the event of getting hurt, but don't let the worry dominate your time spent with your horse.