I think if the horse is looking for you to visit, I would go over and visit just for a minute then go away again. You want him to associate you with reward as well as control. Its ok to call him, but if he doesn't come when you call then you're in a bad situation because you don't have enough basics with him to "make" him come to you.
I would continue to work towards being able to be close to him. That means getting in the paddock with him and eventually being able to pet him all over. However, that can get sticky because you also at that point need to be able to recognize behavior/movements that indiciate disrespect and immediately deal with it. That's where you need someone to help you out. :(
Maybe instead, you could start standing beside him and scratching him, but with things in your hands: halter, lead rope, bridle, blanket, pail, whatever -- so he learns that whatever you have is OK because YOU have it. The first while, he might not like it - especially if he recognizes the halter and lead and associates those with bad things. If that happens, then carry the things with you when you bring out the hay.