I was a lesson a week person for years and years. I bought my first horse at age 40, a 4 year old OTTB. Can't say I ever had a problem with speed within the confines of a ring, but out hacking or on trails, I always had butterflies in my stomach expecting something to go wrong. I never had the carefree impervious days riding as a kid. I established a few simple rules for myself.
1. Don't do anything you're not comfortable doing.
2. Keep pushing your limits even if only by a tiny bit each time we go out.
Between sticking to those two items and finding out you can fall off a 16.2 hand galloping horse and actually not get hurt, in 3 years I've evolved into someone who used to only want to walk on trails to the nut who wants to do nothing but run. I think a lot of that fear comes from some logic that we're "supposed" to be doing something we're not doing. Just go at your own pace. I'm sure one day you'll ride into a big green field and feel like running like the dickens!
That's IT!!!!! The one factor that jas changed since I started having this anxiety was the fact I no longer have an arena to ride in.
When I was taking riding lessons, my instructor had a converted machine shed to ride around in. Now I am riding with an instructor that has no space to go faster than a trot except out on the trails in the pastures.
I will have to keep your rules in mind on my next ride.