Sounds like your horse does not like the cold! I had a horse, Percheron/Arabian cross that was a great ride when it was 80 degrees, ok at 50-60 degrees, and a holy terror below 32!
He got much of the winter off, and we would start back serious work in February to get back in show shape. He had lots of turnout time, so never lost all his condition. I also lunged him regularly in the winter, almost never during show season.
One thing that really helped my gelding was riding on the cold days with a wool blanket under the saddle and pad. It may look a bit funny, but he was happier, so our ride was more pleasant
Living in the south, I don't deal well with the cold either, so I ride pretty bundled up! Thermal or silk leggings under my breeches or sweat pants (yeah I ride in sweat pants when its cold) and insulated boots. When its 40 degrees
So yeah, if my horse is cold, he gets a wool blanket to wear
Oh and LOTS of lateral work! Have to keep his brain engaged...