Gelding getting dangerous with barn/buddy sour issue
My gelding boards with 1 other older horse in his pen but he is able to see a lot of other horses around him. During the day, the older horse is put out to pasture where he cannot see her but she is put back in the evening.
Anyway, sometimes when I take my horse out he tries calling out to her or the other horses, especially if they start neighing but he usually kept his focus on me. When we are in the arena near the barn, he is usually pretty good. But as soon as I try to take him out by himself, he fights me the whole way, is unfocused, and is becoming dangerous. Yesterday, he refused to go forward at one point and half reared. I finally got off him and walked him where I wanted to go and as soon as the barn was hidden by trees, I let him graze. That was probably a mistake because he was still looking toward the barn. Then when we started back to the barn, he started fast walking and trotting in place when I wouldn't let him go faster. I stopped him and made him stand still but all of a sudden he backed up really fast, spun around and half reared again. I pulled him around and made him stand still again. When I was satisfied with him standing, I nudged him forward but he still was in a hurry. I made him do figure 8's/serpentines back at the barn but he still wanted to go back to his pen. Then I tied him up for about 15 minutes, still saddled.
It's starting to get so bad that even when we are in the barn area, where he can see the other horses, he is still pawing the ground when I make him stand still, he keeps neighing, and he is starting to disobey commands.
Where I board, the place is set up so basically anywhere I ride, he can see the barn/buddies but as soon as we get "too far" he starts freaking out. And that "too far" is getting closer and closer to the barn, even when I work him hard around it and let him relax away from it. Any suggestions on what to do?