I'm still afraid of horses and I've been riding since I was a small child - really when I was still in my mother's uterus to be frank. I think a healthy fear is a good thing. Even with my big guy who I adore and I seriously doubt would ever hurt a flea I'm very careful in his stall. I know that with one kick from him I could easily be maimed or dead. He's that big and that young and that powerful. One trainer guy was shocked to his knees almost when he found out I ride my horse in a bitless bridle.
In my line of work, I always tell those I'm training to "Always, always, always trust your gut." If your gut tells you this person is going to be dangerous, listen to it. It's there for a reason. Don't second and triple guess yourself because you could have been right. Better to be safe than sorry.
That said, there comes a time when you have to distinguish between fear that "this horse is really big and things could get ugly" and the intuition that "things are ABOUT to get ugly." That's what I mean with trust your gut.
In addition, horses represent risk. On the pages of this forum and other horseback riding forums, and skydiving forums and motorcycle riding forums you are going to meet people who are generally high sensation seeking. They get off on taking risks to some extent. Since many horse people are women and most women are more risk averse than most men I think you'll find there are an awful lot of people out there who believe being around horses is a calculated, manageable risk.
That leads me to my final thought: trainers, education, guidance. I am not a horse expert. I don't know what to do or how to handle situations with horses. I'm not really that worried I'm going to get hurt - although I have been hurt fairly badly - I'm more worried I'm going to hurt them by teaching them the wrong thing. Having a trainer, a mentor, a friend who knows more about horses than you do, heck even someone younger that knows horses more than you do, can help you to normalize certain situations. As in "see he's not a fire eating dragon." Or as in, "yep, that one looks like a fire eating dragon."
In the final analysis, what I've learned about riding horses off the sides of mountains in the dark on a rainy night is that there comes a point where you just have to "trust the horse." And those are some of the most delightful moments in my life. But to know if that particular horse is the one you should trust and/or in that particular instance you might want to find a professional or seasoned amateur to help you out.