Does he obey you when you are working with him on the ground, or is he constantly wallering you around to see other horse, or whatever has caught his eye?
If answer is no, then more ground work, to reinforce that you are the one running the show.
And when he drifts towards fence to see buddy, make him circle, and make him go the way you want him to go, not the way he wants to. Stop and make him stand still every so often too, both when leading and riding. That again, will reinforce that you are the boss.
Horses make me a better person.