Do you do any bonding work with him, for example: stretching with treats, just hanging out in the pasture/arena together, trail riding, talking to him, etc? He may think that all you want to do is ride/work him? I'd suggest having some bonding time several times a week and surprise and delight him, by sometimes having a treat and sometimes not...just to get his curiosity going and to encourage him to come on his own. Good luck :)
John 14:6 - Jesus said unto him, "I am the way, the truth, and the life: no man cometh unto the Father, but by Me."