Everyone has their own opinion on blanketing, but if it was getting that cold and my horse didn't have a heavy coat, I would definitely be blanketing. Is he outside 24/7? Does he have shelter? As for blanketing during groundwork, it depends on what you're doing. I keep a lightweight fleece blanket on him while grooming, and use a quarter sheet at the beginning and end of our workouts to keep his muscles warm.
In PA, it's averaging in the 30s at night, and low 50s during the day. Zee is out from about 8:00 am - 3:00 pm. He gets a stable blanket at night, and a light turnout sheet during the day. Once it stays in the 40s during the day, he'll go into a medium weight turnout blanket during the day. However, Zee is 21 and doesn't have an extremely heavy coat. A lot depends on your particular horse and his living situation.
I've spent most of my life riding horses. The rest I've just wasted.