I read a book a while back that had some amazing (and super simple) advice for this kind of situation - I've used it several times and it works like a charm.
First, treat him like you're a horse as well. I know that sounds silly, but if you get him to think of you as another member of his "herd" rather than an outsider to it, you'll be more successful in getting him to trust you. A great trick is when you greet him, instead of talking to him or trying to touch him, blow a puff of air in his nostril. This is how horses greet each other. Sounds simple, but I've used this trick on horses that won't let anyone else near them and it's immensely helpful.
The second thing is: ignore him. Go into the pasture/roundpen/paddock with him and do something that interests him (maybe peel apples or unwrap peppermints or splash in the water trough or even just read a book and turn the pages noisily - something that will grab his attention without scaring him), but don't pay him any attention. He'll get curious and come over. Do this a few times, staying for 5-10 minutes once you've got his attention and then leaving without trying to touch him. Soon you should be able to brush up against him or lean on him a little (like another horse in the pasture might) and you can slowly work this towards being able to handle him normally.
Patience and reading his body language is key. Good luck and let us know how he does!