The horse is too thin, and at this point I wouldn't be exercising him until he gains weight back.
It's not just picture angle as we can see spinous processes over the withers, the point of sternum, the neck is very thin, the hip points protrude, ribs show all the way back to the flank, no fat at all behind the shoulder, etc. Probably only a 3 or less on the body condition scoring chart.
If the oats were cut back without adding the same amount in pounds of hay, that would be the most likely cause of his weight loss.
It sounds like your horse is willing to eat more, so I'd recommend giving as much hay as he can eat.
I'd also wonder how many pounds of oats he was getting. Just recently I foundered my own thin horse by giving her too many oats. As the percentage of oats increases in contrast to the amount of hay, ending up with an acidotic gut becomes more and more likely. Then the horse can lose weight due to discomfort and lack of ability to digest properly in the large intestine. It can also cause colic and laminitis, so it's really only safe to feed a small amount of oats to horses.
I was very worried that my horse would lose weight when I took her off the oats, but she has compensated by eating more hay. But in order to do that, the horse needs to have hay available to eat.