Quite a few years ago I had a big wreck on my horse. I had been riding fearlessly since I was 14. I learned fear really quick. I was afraid until I saw Clinton Anderson on RFD TV. I bought his Riding With Confidence DVD series. Unfortunately he has changed things and has it all in one expensive package that the average horse owner can't afford now, but I lucked out and got it when it was available piece by piece. Anyhow, one thing I think will help you to begin with is teaching your horse the one rein stop. It is your emergency brake. Once he learns this, he will stop no matter what. You will gain confidence by knowing you can stop him whenever you feel uncomfortable. Start on the ground with the halter and lead rope, or bridle. You must do this on both sides. Stand by his belly, and pull the rope or rein back to his withers. You must not let go until he gives you slack in the rope or rein. If he dances around, you must follow him without giving him slack. When HE gives YOU the tiniest bit of slack in the rope or rein, drop it immediately. That is his reward. The faster you drop it on his give, the faster he will learn what you want. He will not give much at first, but soon he will be putting his nose to his belly with the slightest touch on the rope or rein. When he is good at it, you will do it while riding. To start with, just get him good at flexing his head around at the stand still like you did on the ground. It is best to reach down as far as you can on the rein, and pull your hand back to your hip. Same thing as on the ground, drop it the second he gives. When he is good at that on both sides you can graduate to doing it at the walk. Walk him forward, reach down and flex his head around. This time you want him not only to flex his head, but also stop moving his feet. Do not drop the rein until he is completely stopped, and flexes. The second he stops and flexes, drop the rein. Do this on both sides at the walk until he is good at it. From there, when you are comfortable, you can do it at the trot. And from there, the canter.(be a little careful at the canter, if you pull too hard too fast, he could fall over.)
If you do it right, you will be amazed how fast he learns this, and hopefully with this emergency brake, you will gain confidence.