Its all good and well to do hours upon hours of groundwork, but if you are scared of the horse, you're scared of the horse! Like riding, there is good groundwork and bad ground work. Bad groundwork is very much detrimental to a horse's training.
You can also do all the ground work in the world, but nothing is going to prepare a horse for going along a trail and having a plastic bag, rabbit or dog shoot out from the bushes and hit them in the face or go between its legs.
Riders need to learn to deal with spooks, to sit them, remain relaxed, and ride on. Without having this confidence, the spook will just increase in intensity as the horse thinks if the 'leader' on his back is scared, then he DEFINITELY has something to be worried about!
This is why having a good trainer is invaluable, just having those eyes on the ground and someone to bark instruction at you, can give a nervous rider a great deal more confidence than riding on their own.