I, too, suggest starting with ground work to develop a better level of trust and respect. That transfers into the saddle! I'd suggest the Parelli 7 Games. Works wonders!
It sounds like he gets impulsive when the other horses go, so the worst thing you could do is hold him back and pull with 2 reins. That only gives him power because he can lean on you and engage his hind end. So the ideal thing to do is to ride with people who are willing to take it slow and allow you to work out these problems. If they aren't willing to do that, then don't ride with them and find better people to ride with! Also ride with people who have calm horses (for right now, be picky about who you allow your horse to associate with). If he starts getting antsy STOP and allow him to settle. Rub him, let him know he's fine, back up a couple steps, then ask him forward. Do this approach and retreat until he's confident and relaxed. If at any time he gets out of hand, reach down with your hand and with one rein bend his head around to disengage his hind end. Do it slowly and gracefully, if you are rough that will only make the situation worse. You can also do a lot of transitions, leg yields, circles if you can, disengaging his hind end, those kinds of things to get him thinking back to you. NEVER let your horse run home, that usually creates an impulsive, hard to handle horse and it's not a good habit to get into. Also, if you at any moment feel afraid and unconfident and think to yourself "I think I should get off" DO IT! There is NO shame in getting off, it's not worth you getting hurt. Listen to your instincts, they will keep you safe. And the old myth that the horse "will have won"...........don't worry about that, it's a bunch of bovine fecal matter ;)