Don't worry, you are just being an adult ;)
When we were young, we didn't think about risk. Now, I get on a plane and think about it crashing. I hate driving on slippery roads in the winter. And I don't like that Harley gets stressed when I ride him outside. I want to trail ride, but he gets pretty worked up about it. In the arena, he's just fine. But I keep taking him out as much as I can, even if it's just around the property. He needs to get over it and I need to stop having worst-case-scenarios of him throwing me flashing across my mind! It doesn't help that when I was a teenager, I had a horse that was terrified of cars and freaked if one went by us. One time, we met a big dump truck and I could feel him getting ready... he jumped into the ditch head first, losing a shoe in the process. Somehow I stayed on. When I took Harley out on our rural road for the first time (only for the 500 ft or so that separate our property from our neighbor's where he is boarded), my heart was in my throat. I never knew that memory had stayed with me that long! A car came and I struggled to keep control. Harley didn't even flinch. Turns out he could care less about cars. He just doesn't like going away from the barn and wants to run back home. And when he starts prancing, and I try to slow him down, and he throws his head in the air and starts to go sideways, I get stressed, but talk to him in a quiet voice.
I don't want this to prevent us from enjoying our riding time together so I've decided I will continue to take him outside and ride him up and down the road ten times in a row if I have to until he just gives up. But what if he doesn't? What if he gets fed up and rears and I end up on the road and my horse is flying back to the barn without me? Guess I'll just have to take that chance...
One of the things I realized that I really enjoy about getting back into horses is the challenges. Every time I ride, I try to improve and every time, a new challenge comes up. Nothing huge, just little things, like holding the two-point a little longer and balancing a little better. So I agree with Cathartes caballus, push yourself just a little bit, but not too much. You will become more confident and more relaxed and in no time, it will be second nature.