This can be VERY annoying. Don't worry I've been there. I just broke Drifter of this same issue.
WALK HIM DOWN. You don't have to chase him per se, but follow him. Don't become impatient or frustrated. Put some good walking shoes on and follow his butt wherever he goes. Eventually he will get tired of it. I once did this with Drifter for an hour. He stays in a 40 acre pasture and it nearly killed me, but I eventually got him. DO NOT give up. Drifter taught me that lesson as well. If you give up and walk away, it will be 10x harder/worse the next day you go to get him.
Now the next step, is find a treat he really likes. For Drifter, it was these peppermint flavored sticks that are the cheapest things ever at Tractor Supply.Practice with him in an arena or such first, so if he runs away it wont be hard to catch him again. Put the halter on. Reward with treat. Take halter off. Put halter on. Give treat.
It took about 3 days for Drifter to realize he only gets these treats he LOVES when I catch him. And I never give them to him at any other time!! Once I put the halter on and we're ready to walk, I give him the treat.
It's made a HUGE difference. He used to see me coming and he would start walking off, but now he'll sometimes even start walking to meet me halfway. Even if he doesnt come meet me, he stands still.
Also, make sure you are not only catching your boy in order to ride him. In his mind that is "work". Take him in just for a groom, and some grain and a scratch every now and then. Maybe even some free grazing. The most important thing I can say is try not let yourself get frustrated and angry as the horse can sense this and will not want to come to you. Also, your mood can affect your body language which you are using to communicate to him so if you were frustrated or angry, it's possible you could actually be driving him from you with body language without even realizing it. Good luck! It will all end up ok, just have patience