Your horse is gravitating towards the fence and avoiding the rest of the arena because he's decided that over by the fence with his buddies is a good place to be, and out in the arena is bad because he has to work and be alone. A common mistake people do when a horse gravitates toward a certain spot is the rider faces the horse out to the rest of the arena and starts kicking them. So the horse learns that if he's facing the arena then it REALLY sucks because he gets pressured hard.
So the best way to fix it is to reverse psychology. Let your horse gravitate towards his buddies, but wave your legs at him and make him work. As soon as he starts to drift back out to the arena, drop all pressure and just relax on his back. Don't worry about steering him, just let him choose where to go. If he makes a
"U" turn back to his buddies, wave your legs at him until he follows the fence and faces away from his buddies again, and then stop and relax. He'll start to learn that being near his buddies is hard work and sucks, and he'll start to look for areas where he doesn't have to work so hard. So you're never telling him to go to the rest of the arena, you're just making it hard for him to be at the place where he really wants to be.