I'm not sure I'm what you'd call a "natural horseman" but this is something I've noticed with herd-bound horses, and something that has worked for me in the past.
A couple things should be addressed here I think.
The horse HAS to be 100% certain that you are in a leadership position in your relationship. There are no buts, and it's NOT debatable.
If he's not paying attention (i.e. Worrying about his friends, fussing, dancing, etc), he needs to work his furry hind end until his attention comes back. I usually do quick circles if I can, back him up quickly, push his hind quarters over. You can make the work constructive, but it's still gotta be work. He doesn't get to relax until his attention is back on you. For me, that means at least one (preferably both) of those ears are glued on me.
One way to establish leadership and trust so that he doesn't argue with you and question whether he wants to be back with his buddies is to challenge the horse with things that he perceives as scary and overcome them. My guy was terrified of those wooden bridges from western trail classes, so we spent some time overcoming that fear with lots of positive reinforcement. You can lay down tarps and walk over them, go for walks with him either on him or off of him, flap scary things around him... the goal is to set up small challenges and overcome them. This will also slowly lead towards a bombproof horse. (Just an added bonus)
Now, to make sure the experience is a positive one, before putting him back in the field with his buddies do something he likes. My guy gets his grain right before going back with his friends. Some horses I know like a really good brushing. When putting him back in the field, maintain that "leadership" role, and make sure YOU are the one who decides it's okay for him to go back to his friends. I turn my guy towards me, make him stand for a moment, then when I remove his halter I push his neck away from me to let him know he's allowed to go.
These are some tricks that have worked for me. Good luck with your guy!