I would have a good equine chiropractor out to check him. I prefer one that is also a licensed vet. My mare started misbehaving one direction suddenly and we figured out that she pulled a long back muscle on one side (likely from slipping or playing too hard out in the pasture). She had a few vertebrae out because of it too. After 3 treatments, some massage, and a few weeks off she was back to new!
I would have the chiro check his back, withers, hips, and your saddle fit.
I would also have someone video you while you're riding. Try to get a shot of you coming and going at all gaits, down the rail and around your turns. When you view the video, pay attention to your hip and shoulder alignment, hands, legs, and your head (where you're looking and if your head is tilted). You could be dropping on one side a bit and not realize it. I had that problem because of a shoulder injury. My right shoulder dropped considerably in the saddle and I had no clue! Once we figured it out and I addressed it, my mare's odd behavior when going one way stopped! It took a long time to permanently correct it, and I still have to self check if my mare acts oddly just to make sure it's not me, lol.
Your saddle could also be off balance. If there is a saddle repair shop near you, take your saddle in and have them check the tree, stirrup alignment (and length!), and flocking (if it's English). If anything is off kilter even just 1/4", it can effect your position and how your horse goes.
If ALL of this checks out, then I would do some work on the lunge line in side reins and work over trot poles. I'll post instructions for both after you have had all of the above checked out and confirmed okay