My horse is/was exactly as you described your horse. He would MUCH rather not ever have to lead if he could avoid it.
When he was first with other horses he would go real slow, hoping and allowing the other horse behind him to pass him.
And almost the whole time he was first he kept looking left, then right, then left etc etc FOR things to spook at. Fun ride, NOT!
The first year I spent just getting to know him and was put in front a lot because the others I rode with didn't want to go first. Needless to say those rides were nerve wracking and he spooked a lot and I fell off several times.
The second year I spent on getting him over this and rode with people who allowed me to work on him going first. We did a lot of the leap frog game and other than that I kept him busy working and concentrating on me while he was first.
This year, year three he now goes first without any problems what so ever. He would rather be anything but first, no doubt about it, but, we have worked on his confidence enough that he will do it safely.
Now on the flip side of things I was always able to take him out by ourselves without the problems we had with other horses.
My theory on why is this:
When he is first with other horses it's a huge responsibility. He, my horse, thinks he has to be the leader, cause he is the leader and he takes that responsibility seriously. He wants to keep alert and vigilant for any and all predators and lead them away fast, or, to get away faster than them so he survives. I am not sure which.
When he is following behind the leader he lets them do all that worrying and even when they spook he has no reaction from it because he sees they haven't been eaten. (Some other horses feed off other people's horse's spooks, my boy never has thankfully).
But, when we go out all by ourselves he has always been MUCH better. He is a bit more alert than if he was following another horse, but, all in all he does real well.
That's because he letting me be the leader and not depending on the other horses on the ride. He is more actively listening to me and putting his trust into me.
This is why when I go out with others I had to step up and keep his focus on me rather than on the other horses and what they were doing.
The more you keep your horse's mind engage while riding with or without a group, the more he's thinking of what you want, rather than what he can spook at.