I don't think that is an uncommon fear at all, especially as we get older.
For myself personally, having kids had a lot to do with any fears that pop up for me. I certainly used to be fearless, and had no qualms about riding any horse, as fast as possible, lol.
After I had my kids, I started wearing a helmet, and became very aware of potential dangers, so became a more cautious rider. Had to protect myself for my kids.
I have no worries riding my old guy, I've had him 15 years, we know eachother very well. I know his triggers, and know what to do with him if he Does get wiggy.
My 5 yr old gelding is another matter. He is generally good, but can explode over certain things. Afraid of the water hose, puddles, flinches if you sneeze, lol. I have only had him a few months, and do not completely trust him under saddle. He is good in the arena, but I mostly trail ride. And there are actually few 'trails' around here, mostly just open fields. So I seem to be afraid that he will bolt, even though he never has before.
Anyway, I certainly sympathize with your situation. I think we should do square breathing!