It is natural to have different fears. It's just part of riding, I think.
Just realize that you are a strong rider and know that you can stay in the saddle.
Acknowledge that falling off is a possibility but don't dwell on that possibility. As a rider, you have to understand that risk is involved but chances are you'll stay in the saddle... that's usually how it is I assume, anyway.
And, it's kind of the same thing how some people find western more comfortable than english, and vice versa. Some people prefer bareback to other disciplines. It's a matter of personal preference. I'm the opposite but hey, comfort varies. Some people prefer a hard chair to a recliner...
God bless, englishaqh (: