Winter is a brutal reality for about 6 months a year here in northern Alberta. We either ride in the cold, or don't ride. Having our horses at home also means we don't have an indoor arena to ride in, so we have to bundle up pretty well for those (-20 C and cooler) days. Here's what usually works for me...
Feet - good quality wool socks (SmartWools are awesome) with tall Ariat winter riding boots. I don't think the Ariats are available any more, but have heard Mountain Horse makes some good winter boots. If my feet are cold, my body cannot stay warm.
Body - lots of base layers. I have a pair of fleece lined breeches which are pretty good, but instill throw a pair of sweats or thin outer pants over too, or a pair of tights underneath. I also have a pair of fleece lined jeans that I wear trail riding. They are awesome. I wear a long sleeved thermal undershirt layered with either polar fleece or wool sweater and a warm winter jacket. I don't know if you have Mountain Equipment Coop in your area but their outerwear is excellent. Otherwise a quality lightweight cold weather hiking coat is ideal.
Gloves - I gave up on riding gloves years ago. They just don't make them warm enough for our climate. I use lightweight, ultra warm ski gloves instead. Yes, they are bulky, but if they are made of the right light materials, they don't limit movement and feel as bulky as you would think.
Head - I wear a thin neck tube called a "buff", that I purchased online, but you can get them at a motorcycle store. It is awesome. I also have a polar fleece ear warmer that is made to be worn with a helmet. Basically it looks like a wide headband that is wider where ther ears go and only goes around the head 3/4 of the way. The front is elastic and sits on top of your forehead.
We ride bareback for much of the winter, using the warmth of the horse to stay warm ourselves. Also, with limited daylight hours and cold temperatures, we don't ride as long or as hard. We do try to get out fairly often though to keep everyone in shape.