if you do not know how to do ground work for a correct response, it will not help this . It will only add to the confusion, actually. Ground work done incorrectly only builds a horse's estimation of his handler as a pushover or not.
I would go back to the saddle that he behave best in, first. Then, do not ever stand around resting and chatting in the middle of the arena. If you must go there, keep him moving, circling. Allow him to rest only standing out by the edge.
If you are trotting around and he falls more and more into the circle, then go back to a walk if necessary, steer him around , bring him out to the circle and trot on. If he stays on the circle for a bit, go back to a walk and reward him with a walk, on the circle, or even a stand, on the circle (*I mean 'on the rail').
Just do NOT allow him to go into the middle, even if you must change gait and put him back out. Over and over. If he goes into the middle, you circle around small, and if you can make him go fast, all the better. The idea being, work hard in the center of the arena, trot or walk nice and easy on the outer part of the arena.
And take care not to end your ride in the middle any more. End on the outside.