Also, you need to work more from the ground to focus him on you, so that no matter what he is doing what you want.
That can be as simple as when you halter, wait for few minutes before walking, stopping during being led to make him stand, and that means stand without you petting on him, talking to him or even looking at him, just standing there.
Vary your speed as you walk too, making horse match you.
During riding, if you can, circle away from and back to the others. It would help to do this at home also, if have someone can do it with, but that will take anxiety off to a degree hopefully.
Also, make sure that when you are riding, that you are paying attention and are aware of your voice getting higher, as that will add to horse being up, and that you keep low hands, and don't ride with people that are going to want to race around either, as that hypes up horses.
All of you would benefit from the standing stock still thing while under saddle too, helps focus horses and reinforces that humans are in charge. Just stand for various lengths of time.
Horses make me a better person.