I find it odd that the wither rub in the first photo would be from him gaining muscle. It looks more like the top of the saddle fork was down on top of the wither bone, and that, to me, would indicate that the area behind the shoulder had actually atrophied and dropped a previously clearing tree down onto the wither bone.
AS for the fit of the Toulouse, I don't know a lot about how that kind of saddle should fit. It looks like it follows the angle of the shoulder nicely, but it looks front high, like it's maybe too narrow? Or , like unclearther said, just lacks the curve needed to follow the horse's back. It looks like it is kind of pressing down into the cantle area, like it's going up a hill. Have you checked for bridging?