We don't get as cold here as other folks but we do sometimes have cold weather where the wind chills go below 0*F. None of my horses ever get blanketed. Providing that the horse is healthy and grows a decent winter coat, it's actually more effective to leave them unblanketed because their hair can fluff up and insulate them the way that it's supposed to. If they are blanketed, then it flattens their hair and they lose that level of insulation.
IMHO, the only exceptions are horses like Anebel's that have been clipped or if the horse is sick or doesn't grow enough winter coat for the area that they live in.
Just for an experiment, next time it's cold, leave his blanket off. If he's shivering, then he might need it. If he's not, then he's just fine.
If you make sure and keep roughage in front of him 24/7, that should be enough to keep him warm.