Yeah, fleece is comfortable, but the hair does stick to it like crazy so I avoid it at the barn or while riding unless I can wear a shell over it, which you can't because of the noise I assume.
Never thought of the hoodie issue! Thanks for bringing it up. I ride in hoodies in the shoulder season, but that's over now! I do like my puffy vest to keep me warm on cool, but not cold days. It keeps my arms free. There are all kinds of puffy vests at thrift stores, but I splurged and got a new, brand-name one. I love it and wear it a LOT! (mostly because it's about the only new riding wear I've ever bought for myself)
I'd look for zip-ups that aren't fleece. Not easy to find, but there are some in sportswear sections. Or just a sweatshirt without a hood. Those are pretty common.
I also get lots of riding jackets at thrift stores. I have some shorter ones and longer ones for colder days which I just open at the front from the bottom (if it has two zippers, you can do this). That way it lays on my legs a little while I'm riding and keeps me warm.
Layering is key of course.