I think you are learning a lot keep talking to your Vet.
Also about other people seeing him, I would if I was you keep very good records on worming, teeth work, vet checks, how much and how often you feed what grain Keep your recites. All this will prove you are not neglecting this horse but that you are working very hard to get him into better shape.
One other thing, do a little looking into getting your hay tested to see just what he is getting from it. Also, do the same thing with your pasture grass. get it tested then you will know what he is getting.
You're not riding this horse are you???