Holy cow, when I first got my OTTB, I felt so badly because I didn't think he'd be able to reach his full (or even half) of his potential with me as his rider. I had been off for over a year with a really bad leg after a fall. Three years later, we're starting eventing, schooling 2nd level dressage moves, jumping things bigger than twigs and running around on trails like a pair of idiots having a ball. My horse is waaayy cuter than me! I tell him all the time.
You just have to see your distance...you don't have to like it.