Theres 2 things you can do 1. Try to relax. I know that's tough but if your nervous the horse is going to be nervous.
2. Try to get him used to scary things. I work with my trail horses with kids on bikes, 4 wheelers, cars, plastic bags/bottles just anthing I can think of to get them used to scary things.
I have to say I have had a horse that was just too spooky/dangerous to be a good trail horse. I'm sure if I was better at training I could have worked out the issues. I'm not telling you to sell your horse but to be careful. If my horse gets scared I put her nose to butt with one that isn't.
It takes miles and experiences to make a good trail horse. They have to be pretty fearless and have trust in their rider. The more miles you can put on your horse, and the more things you can expose him to the better he will become.
Edit. I want to add that the horse I mentioned above an OTTB was excellent in the arena, had perfect ground manners and was an excellent jumper (although I never jumped her

) I tried for 2 years to make a trail horse out of her. She never could relax in the wild wide open. I finally had to sell her to a riding school.