If you can't ride him out with confidence, try leading him out. If walking a horse is what you can do now, accept it and build on it later.
I second this approach, as a good technique in general (not as a specific prescription for the OP's situation; there's more than one way to skin a cat) - and make sure your horse doesn't come loose or have mechanical advantage over you if he spooks while you are leading him. You've got to be able to hang on to the horse. Ground work first - in familiar environments - before gradually taking him further out of his comfort zone.
One simple trick for hanging on to horses better on the ground in touchy situations: Loop the lead rope over the nose - threaded through the halter - but make sure the metal clip doesn't dig into the jaw; if that happens, use a clipless rope and make an appropriate knot instead (e.g. bowline). I've not had horses get away from me with this
arrangement, even when I was a child - here I am below at age 10 with a horse who was notorious for wanting to take off on you when spooked:
The rope trick works as well for me as a stallion bit - make sure the rope sits correctly where the nose band of the halter should be - not so far down that it's on the soft tissues - but not so far up on the face that it confers no mechanical advantage to you either (you can click repeatedly to enlarge the photo to see the relevant detail). Having said that, I'm not afraid of horses and I know where to stand to have mechanical advantage over a horse on a rope. Stand in the wrong spot, and you're water-skiiing on land. An angle of 45 degrees to the horse's head works well, either in front or behind. That way, you make him slightly off-balance if he pulls.
I grew up helping to educate harness horses, as well as saddle training my own riding horses from scratch. For harness horses, led and long-rein work is standard before they are hooked into a cart, and the technique we used was to have a "babysitter" at the horse's head, on the lead, and the other person behind, while teaching long-reining, or later when hooking up the cart etc, until the horse knew what it was supposed to be doing and was confident.
I was usually baby-sitter, but sometimes I was in the cart as well, such as here, as a teenager, with a yearling Standardbred filly called Classic Juliet.
The babysitter technique works very well for initial harness training, and also for the horse's first 10-15 minutes with a rider on its back. Harness to saddle is low drama this way - after 10-15 minutes with a babysitter, I'm generally riding the horse independently in an environment with which it is familiar - in the case of harness horses, I did their initial riding on their home training tracks, and then ventured further out.
The filly above had a colt called Classic Julian after her racing career (birth family's best race mare), and he also raced successfully. When he retired, my husband and I adopted him a couple of years ago. He now lives on our farm, with our other horses and donkeys, and I'm now in the early stages of re-training him to saddle. The main preparation I do is to lead him all over the local trails where he's going to be riding later, so that he's confident in the area. We walk him when we walk our dog, and it's great fun for all of us:
You'll notice he's not in a nose loop in these photos - he doesn't need that anymore, he's so cool and calm now. In the photo below, I've let him off the lead, as I often do when we're back on our own property at the end of an extended walk. He's then free to walk beside, behind or in front of us. As we're his "herd" this is just what he does. Here he is walking ahead of us - and because he's a harness horse, he still feels like he's "in touch" with me there - as this is a normal place for a handler of a harness horse to be.
Often, other horses and donkeys will tag along with us too:
Julian is now such a cool cucumber around our local area, with cattle, machinery, kangaroos, emus, crackling in bushes etc, that I won't have to worry about him spooking when I start riding him on these trails later this year.
Sunsmart was the same - I re-educated him ten years ago, and he's a very relaxed horse to ride - after a rather spooky first half year, when he was first in the "real world" post harness track.
In case you see a resemblance, they are both by The Sunbird Hanover, a son of the famous US pacer Albatross.
However - I totally agree with
that you've got to work on your own fears first. If you're nervous while handling or riding, it's going to be hard to make progress - you'll be infecting your horse. Work in your comfort zones, and slowly stretch out from there. Good luck!