I wear Terramar under clothing, it's similar to UnderArmor. I also wear my Boulet boots that are too big with wool socks. That's if it's in the twenties and thirties.
If it's really cold, 10* and below, and I'm going to be out horseback for a long time, I wear: My under clothing, sweats and no jeans, wool sweater, a down vest, insulated coveralls, a pair of thick wool socks (I have worn battery operated socks when I was going to be out for just a couple hours - heaven), my Sorel boots which seem to be warmer than the Muck Artic Sports I bought and they fit my winter stirrups better. Wool glove liners and some gloves that are promoted as being for hockey moms. lol I also wear a wild rag and either an Elmer Fudd type cap or my western felt, if it's windy.
I pack extra gloves, mittens and socks in my saddle pockets along with matches a small, insulated container of warm water, flares, stuff like that.