[mls]

If your horse is walking away when he sees you with a halter, he's come to associate the halter and work.

Or the horse is sore, fatigued, unhappy, bored, etc.