All my horses know when I walk out in the pasture I am the boss, I have 4 and 1 boarded here. This summer they were up at a neighbors on their pasture and another neighbor whose horse had been with our's was up there, they put him in with our's.......my appy gelding was being very agressive with him and the neighbor tried to catch anyone to put the one neighbor's out but they all ran off on her and wouldn't let her catch them! She called me to come help get the one out.....I drove into the yard and yes they were all giving the one a hard time, I walked to the gate, hollar "HEY" and they all stopped in their tracks with this quitly look LOL my neighbor said she was surprised they didn't all line up to salute me LOL
If I were you I would definetly have something with me when you go out into the pature and let him have it if he so much as looks in your direction with a sour look! That is just too dangerous!!