If your horse is doing fine without grain (as in, good condition, not losing weight) then I would stop giving it. My vet says that hay/pasture is their food, grain should be more like dessert. (You might want it, but you don't really need it,)
Our horses sound like they were in similar condition when we got them. Thin but not bad overall. We fed grain to help build weight as we were going into winter when we got ours. Since the grass is up and lush we have cut the grain down drastically. It is now more of a treat and almost a ritual. We let our horses out of their pen every morning and call them back in and give them about 24 oz. Each of grain in the evening. ( we use a 12 oz. Cup to scoop with.) Its not a lot and they of course love it and it helps us to keep our routine.