A long while ago I used to do as you do, until I realized that it was unhealthy. Why am I getting upset with my horse who only wants to please me and enjoy being out with his friends? He wasn't the one that was xyz (some negative experince) to me that day. And even if he was, I need to just get over it and stay neutral.
Now the barn is my place to wind down and 'get my head in the game' instead of worrying or being angry.
If I'm to a point where I'm feeling so negative then I don't even get on. I just spend time relaxing with my horse, pampering him via grooming or a trail walk. I always leave the barn in a good place.
"Strength is the ability to use a muscle without tension"