Agreed, get rid of the sweet feed
Is he wormed regularly? Have his teeth recently been checked? The first two things I would do is have a vet do a fecal egg sample, even if you regularly worm, and have a vet check his teeth.
Get something(s) that are high in fat and fiber. I would recommend beet pulp, good amount of fiber, and if it has molasses in it even better for gaining weight, and rice bran, it has a good amount of fat, and as a bonus, gets their coats shiny and dappled. In your case, I would probably recommend about 2 lbs of Rice Bran, and maybe 3/4 - 1 lb of beet pulp, at least if you would get the shredded kind, but it is be a good idea to talk to your vet and see how much he feed she/he recommends and to what amount.
I would definitely recommend keeping hay 24/7 in front of all
horses, especially hard keepers, but if you have good pasture, you probably won't need the hay until it starts raining/snowing.