Not sure what to do

Wow! I can see why you're having trouble! I agree with the others that this is going to have to be a two step process to get him building topline and then finding a saddle that will fit him down the road. In the meantime, I agree with the suggestions on lunging him and using corrective pads. I don't know how you normally ride (sounds like western?) but you might be able to get a cheap used Bates or Wintec english saddle with the changable gullets to use with the corrective pads in the mean time while you work on his muscle. Those can have narrower gullets than any of the western saddles I've seen. Not saying they will be a perfect fit, but they may be the best option, along with the pads, in the short term to get a reasonably close fit while he continues to change shape.

He's on very fresh hay 24/7 right now.
I plan on upping his feed to a higher fat for the winter.
He's also on grass a lot too.

I personally don't think he's skinny. Just lacking muscles+top line

I don't think he needs more feed - he needs to work. I agree - he's not skinny. Plus he used to be overweight and you definitely don't want to go that route again.