If your horse is doing fine without grain (as in, good condition, not losing weight) then I would stop giving it. My vet says that hay/pasture is their food, grain should be more like dessert. (You might want it, but you don't really need it,)
Our horses sound like they were in similar condition when we got them. Thin but not bad overall. We fed grain to help build weight as we were going into winter when we got ours. Since the grass is up and lush we have cut the grain down drastically. It is now more of a treat and almost a ritual. We let our horses out of their pen every morning and call them back in and give them about 24 oz. Each of grain in the evening. ( we use a 12 oz. Cup to scoop with.) Its not a lot and they of course love it and it helps us to keep our routine.
Be strong and courageous, and do the work. Don't be afraid or discouraged by the size of the task, for the Lord God, my God, is with you. He will not fail you or forsake you.