I ride all winter and in Maine that sometimes is COLD. Dress warm and most important remember to keep feet warm as there is nothing worse than dismounting onto frozen feet! I use foot warmers in my boots.
Of course we don't go out if it's freezing rain, or icy as we don't have an indoor arena. You have to be careful of icy ground, and when the snow gets too deep we will stick to the edges of the field, or the ring, which we plow.
Winter is a great time to ride. Enjoy!