You say he is the only horse on the property at the moment, this is not helping the situation. Horses are herd animals. I think a horse should have the company of other horses, at least be able to see another horse in a paddock beside theirs.
That being said, you should be able to take your horse away from other horses to work with it.
When I got Charlie, an ottb, he was a fence pacer, he could pace the fence line all day long. It wasn't because there weren't other horses around, there were plenty. I have had him for about 15 years now and he doesn't pace the fence line anymore. He did do it for a couple of years though, I did my best to try and figure out the reasons for why he was doing it. I never did figure it out, but he did stop. It can become a really bad habit. I had no issues whatsoever with taking him into the barn or riding him when there were no other horses around.
If I were you, I would first of all get him another pasture or turnout buddy, or at least have another horse in a paddock beside his. I know you are trying to work on him not being so dependent on the other horses, but you really need to work on getting your horse to first of all respect you and depend on you and not his buddies, even when you take him away from them.