He probably will not "want to be with you" until long after he learns to trust you with the things you ask him to do. So, the advice from MLS will help him learn to tolerate and trust you. Just like one of the other posters said that their horses are sensitive to sudden actions, etc., one of my horses does not "want to be with me" unless it's feeding time, and then he doesn't like to be touched. He tolerates it, because he has manners, but he does not crave human contact. I've had him for 30 years, since he was 11 months old. So, don't beat yourself up if your horse is like that, too. He may never be a pocket pony, but if he turns into one, it will be after the trust and respect is there.