So think about what it is you want him to learn from 'join up'. That you can make him move, and that it's easier - & safe - to approach you when you ask? At the moment it sounds like he's got the idea that to look at you will 'switch off' your pressure. But if he won't come on his own, due to prior experience, you can just use the same principle of putting some pressure 'out behind' him. If he leads OK, I think it's easier to teach that way IME. You can use less pressure & be clearer.
So... he's facing you & you want to 'encourage' him, so you invite him(different bodylanguage to telling him to leave) & if/when he doesn't, throw a bit of pressure out there to get him to move his feet. He naturally won't understand this means come, so will move in the wrong direction. But if you have a lead on him, you can use what he already knows to back up your new 'vocabulary'. As with the rest, reinforce any steps in the right direction by instantly quitting any pressure, which may well even include facing him & standing your ground to begin with.
I probably wouldn't even try to pet him, until he's confidently approaching & staying close to you when you ask. Sounds like he will allow handling if he knows he must, but has learned it's not pleasant or it's dangerous, so if he has a choice... So I'd definitely work on this without restraints & very gradually, if you want him to learn to enjoy being touched, rather than just tolerate it. I'd also personally make the most of your relationship with your other horses, as rookie suggested and personally I'd also use food with him.
I think positive reinforcement(rewards, Good Stuff) is very important in training, to teach the horse to actively enjoy our 'games'. Negative reinforcement(release of 'pressure', removal of Bad Stuff) IMO is vital to horse training, but I don't think it should be the 'be all'. With most horses you'll find at least a few things they desire from you - a scratch on a sweet spot for eg - but with horses that have learned that humans=discomfort, food treats may be the only thing for now you can give that he may think is good. If you're touching him while he's choosing to stay & eat, if he's not stressed(that may mean very gradual with him), he will begin to associate your touch with those Good feelings.