Welcome to the awesome world of saddlebreds.
It sounds like you did a good job getting the weight on, but now that he is ....fat, have you reduced his rations to a 'maintenance' level?
If your beginning to work him now, I would wait and see. Saddlebreds can be a little hard to keep.
Working him is the only way to get him fit. Just like people you can get inshape sitting around. If your fearful of trying under saddle now, keep working on the ground. Lunge him, poles on the ground, ground drive him, anything to make him move.
Once a saddlebred does trust you, they will do anything for you, but it takes time. These guys are much smarter than most breeds and can take a little more time to trust. Just hang out with him with no expectations sometimes. He will figure out your a friend that just wants to go adventures with him.
Pics???!!! I love saddlebreds!!