I'm in Minnesota, so I know the cold you're talking about! I don't ride Clem in REALLY cold days, the days where you go outside and the air is so cold it stings your throat and lungs, because if it's that uncomfortable for me working the horse wouldn't be so comfortable for her, either. But I make sure the footing is secure, no ice under the snow. Wear LAYERS of clothing. Fleet Farm down here sells flannel lined jeans, you could maybe order some online. Leggings and underarmor. Big wooly socks. Winter boots (Sadly I don't have winter riding boots, so my feet freeze.) Maybe even get those footwarmers and stick them in the toes of your boots.
On top you want layers but not so many that it impedes your movement. Undersrmor, sweatshirt, heavy jacket. I have an old german army coat I use - it's made for warmth, and it's very long while the zipper only goes down to my waist which makes for easy movement. And it's machine washable. I bought a VERY warm and VERY heave leather jacket lined with soft, but it's dry clean only. So if I wanted to wear it anywhere else I would have to dry clean the horse smell out of it :p
If you wear a helmet, wear an earflap hat under it or your ears will get COLD. If it won't fit under your helmet, maybe invest in a slightly larger winter helmet that a hat will fit under. A scarf so the wind doesn't frostbite your face. and definitely gloves, or you won't be able to use your hands after 20 minutes. Believe me, if your hands, face, or ears are cold you will be miserable the whole time. Someone mentioned those chemically heated back things - I've never thought of that, but it's a good idea. I know they exist for back pain - get one of those, and wear it underneath your clothing to keep you warm.
As for the horse, if it's cold enough, I know they sell blanket things that you use while riding that covers the back half of the horse to keep it warm. If you don't blanket in the winter though (I don't myself, Clem grows a massively thick winter coat and has never needed one) that wouldn't be necessary. Warm the horse up a bit longer than you would, let its joints warm up and get the horse a bit warmer. Warm the bit up so it's not so shockingly cold.
Riding in winter, while dressed correctly, can be beautiful - but believe me, I know your pain with the cold winters. Going out to visit your horse in -35 degree weather when the wind is screaming is NOT pleasant. Gotta wait for the non windy days at least. The plus side is if there's a lot of snow, if you get dumped you have a cushion!