If you overuse the one rein stop? It will cease to have any effect at all.
This horse has some serious issues with you as rider/handler and those need to be fixed. The fact that he is not doing it with others, tells me it is more you than him.
Take some time and look into how you handle this horse, leading, feeding, haltering/unhaltering to see if you are telegraphing to him that you are lower than he is.
And while working him when back at barn might seem logical? To the horse, he has won, because where is he? The barn, where he wanted to go anyway. And working/leaving/working/leaving will only make this worse I think, but might not.
However, as to him associating food with barn, until you get some control over him, this is not going to stop. And the fact that he has drug you off using fences, if I read OP correctly, means you badly need to get more instruction in your handling techniques.
And too, this horse doesn't respect you enough to put up with in/out routine suggested above by several posts, and I think this is a recipe for disaster, considering horse's ability to win this battle already.
Horses make me a better person.