I never try to hide the halter or rope. If I have it, I want him to know it. Then again, I also NEVER go into the pasture without it, so he doesn't know if it's a "be haltered and brought in for grain" sorta day, a "be haltered and worked" sorta day, or a "be haltered and given a cookie and then set loose again" sorta day. Or even sometimes even just a "Hey, I'm here catching up another horse entirely, Go Away!" sorta day.
Makes it easy when he knows that I always have the halter and he always will have it put on, but that it doesn't always mean things he doesn't like are going to happen.
Sounds like you go out and pat him, yes? Do you got out and feed him or bring him in for food? I would make the halter just part of the routine. If he wants food and/or attention, then getting haltered is part of the package. Now, if he's on 40 acres of grass, he might not care enough to change his behavior.