I'm going to focus on this part of your original post, as I think other posters have addressed the other issues. My overall impression is that your horse was never truly "reclaimed" from the track. Saddle fit may also be part of the problem, but I think the biggest issue is that your horse really doesn't know how to be a riding horse and is stuck in driving horse mode.
Have you ever truly watched a driving horse in shafts and how they turn? They don't bend their bodies, because they can't - the shafts prevent them. They turn by holding their bodies still and pivoting on their hind ends, crossing over with their front legs. So when you ask your horse to turn as a riding horse should, bending his body and making a gradual, turn, he can't do it. The fact the he comes back to a walk in order to turn tells me he's a smart, nice guy who wants to live to see dinner. His reaction to being unbalanced is to slow down - that's a good thing.
Teaching a horse to bend and balance around a turn is 1.) a job for an experienced rider or one supervised by an instructor and 2.) more complicated than I can explain in a internet post.
Of the problems you mention in your original post, most are attributable to the horse's history on the track and not being reschooled as a riding horse. If you can find someone to help you, even for a short time, that has some experience reschooling OTTSBs, that would be best.