Basically IMO cereal grain is problematic to some degree in most situations, so even if a horse requires extra calories, phosphorus, whatever, there are generally healthier options IMO.
Lots of folks believe that horses need some sort of ration balancer to maintain diet health
Ah, but I would call your loose minerals a 'ration balancer'.... assuming they appropriately balanced the grazing of your specific situation
I only feed on days my horse gets worked hard or in the coldest parts of winter
I think it depends what you feed, but the main problem with this regarding grain is that it is difficult for a horse to digest and process anyway, especially when not fed little & often. The horse will get little benefit and it will enhance likelihood & degree of health probs due to it's feeding. If you're feeding because a horse needs more calories, feed little & often.
If calories & nutrition aren't an issue, keep occasional treats very small(eg a handful) & if grainy, ensure it's well processed &/or only oats, not higher starch/less digestable grains. Also I wouldn't feed high starch/grainy feeds at all, even as a treat, if there was a likelihood of the horse having metabolic or gut issues.