Now, I'm about the biggest scaredy-cat out there. Horses, to me, seem to be one of the safer sports, when approached with a bit of caution. I find that the fact that there is a live animal that weighs ten times what I do involved makes me more careful. I wear a helmet every ride, I check my tack for problems while I tack up, before I mount, and as I untack and out things away. I make sure I have control of the horse on the ground before I mount, etc. Most of the statistics about horse related injuries/deaths include not only experienced riders/trainers on seasoned mounts who encounter unavoidable accidents, but also the drunks, the inexperienced in over their heads, and the plain idiots and people taking stupid risks.
I could be killed half an hour from now, hit by a bus on my way to class. Shoot, I could be seriously injured falling down the stairs to get outside to meet the bus!
You look both ways, you hold the banister. Use common sense and be safe, but if you cut all risk out of activity you'll end up sitting in a dark room sipping weak tea and living on vitamins for the rest of your life.