Thanks for all the thoughtful responses. I kind of felt like I had failed him when they cornered him in the arena and worried about what message he understands from that. How can I be his leader if I can't keep those other horses at bay? But how does a 5 ft 125 lb person stop three charging horses (one of which is a Percheron X)?
I think I will make it a habit of going in with the lunge whip, even if it's just to go in and give him a pat. I was in last week to take measurements for a saddle and since he's so calm, figured I'd just do it in the paddock rather than bring him in and again, a horse charged at him. Next time, I will try to establish a circle of safety around me and try to keep the other horses away. But they can be fairly aggressive, even charging behind us. Today, when one of them appeared to want to do so, I draped the lead rope over Harley's neck and walked a few feet away from him and wagged the lunge rope at the mare who was coming behind us. Harley stood as still as a statue and she moved away so I was able to continue to lead him out without issue. But these horses are not respectful of people.
Also, I'm not sure how the BOs will feel about me getting harsher with the whip, however, since they take a very laid back attitude to it all. Maybe I should just have a conversation with them about it so they know why I am hitting their horses with a whip if it becomes necessary. They are my neighbors and have been wonderful BOs, so I don't want to do anything to damage the relationship.
As far as moving Harley out of that herd, that's not an option. There is no other herd, this is a small private barn with only three horses, Harley being # 4. Oddly, the horse that shows the most aggression is a 30 yr old appy gelding! None of us expected that. The good thing is that the old guy doesn't have enough energy to run very fast so Harley just kind of prances away from him. But the other two now seem to be ganging up on Harley.
sarahfromsc - Good to know Harley's not the only loner! Honestly, he sometimes seems to prefer people to horses. I talked to someone about this recently, and she was saying sometimes it's because horses weren't socialized a lot when they were young. It makes him a highly trainable (and lovable) horse, but I can't help but feel for him when they all go at him like that. We will be moving him to our property next summer and getting a companion/mom horse and I swear, I will find the most placid, accepting, docile horse ever for him! He does seem to do better at one-on-one relationships than herds.
Woodhaven - my daughter is nowhere near these horses. I would NEVER allow her to come into the paddock or pasture as these horses are far too disrespectful. The only thing I allow her to do is close a gate behind me as long as there are no horses nearby. And actually, that's often how I handled it in the past since there are several adjoining paddocks so I could usually isolate Harley and get him out safely, but with winter setting in, that's no longer an option. I feel I need to just let these horses know that they need to stay away from me. They're not aggressive towards me though, they just walk up to me and sniff me, but I'd rather they didn't because I can't have them following me right up to Harley.
Alhefner, I like your idea. I may try that. It does make me a bit nervous though, because there is a bit of a bottleneck around the arena door. The paddock basically narrows to a passageway that leads to the arena. But there is also a gate there, so I suppose I could push them away enough to get the gate closed, then open the door, then push them back some more before opening the gate. It's more work than letting him in through a side gate, but maybe more effective at sending a message - to both Harley and the herd - that I'm his protector.