Because of my work schedule, I need to ride in the mornings. The barn where I ride has an unheated indoor, so my personal temperature cutoff is 25F outside (indoor is typically 15 degrees warmer). Below that, I don't go out. When we're close to the freezing mark, I ride with two layers of socks, fleece lined riding tights, and fleece lined gloves!
The biggest frustration is when the path out to the turnout paddocks turns into a slick sheet of ice- it can be really scary bringing them in. I also typically ride for shorter periods of time, as you really have to build in the time for a thorough cool-down. The mare I ride was clipped last year because she got so sweaty even just doing walk/trot work, but she's not clipped this year so the coolout has to be even longer. With limited time, it is sometimes a balancing act to decide if the amount of time for actual work will be worth it once the time for coolout is added on.
Lucky for us in New Hampshire, we've really had a very mild winter so far. A few mornings have hovered right around my temp cutoffs, but I actually worked in the outside paddock on Saturday because it was bright and sunny and in the 40's!