I too had a herdbound horse and I can relate to the frustrations. For me, my horse and I just needed to build up trust. I know this sounds simple, but what I do is keep a whole handful of treats in my pocket, and just walk him around by hand and slowly move farther and farther away from the other horses, giving him treats when he stops and settles down. You want him to start to associate good things with being out alone- it's a long hard process but it's pretty manageable with patience and baby steps.
I make a point to work my horse (a good hard workout) alone in the round pen where he is able to see the other horses, but it's still me whose in charge. Then his cool down is me leading him (not riding him) through the trails with treats and lots of petting. I'd also recommend working with him alone, and not just trail riding with other people- you want him to respond to you, not just follow the other horses.
I know how hard it can be! Just keep your chin up and stay calm around him- if you relax, he'll feed off that and start to relax too! Hope this helps.