To the OP, if you were just venting, then you have our sympathies. If you were asking for advice, then here is mine:
If your horse is used to being rugged every winter at her previous home, then you should consider rugging.
If she is lacking enough weight to keep her warm then you should consider rugging.
If she is losing weight because she is cold then you should rug.
If it rains non-stop for days on end and she has no shelter, and is soaked through to the skin, then you should be rugging.
If she is clipped in winter then you should rug her.
If she is healthy and happy, keeping on weight easily, then you do not need to rug.
Other people often attempt to offer advice in a 'casual comment' manner because they don't want to be preaching. They don't realise that everyone is saying the same thing. Instead of getting mad about it, engage them in conversation - ask them why they rug, take the opportunity to learn. We are all learning all the time