I agree, he has got to respect you rspace, which may require getting after him a bit. That said, be aware of where you are. Should he react negatively towards you getting more forceful, you don't want to get cornered by him.
You could also consider sweet talking him---if he is food motivated, consider giving him a treat each time you go to his stall to get him. If he responds well to this, consider just working with him for 15 or 20 minutes each day---go in his stall, give him a treat while you put the halter on, take him out of his stall, then praise him (petting, a brief grooming or walk around, something you know he likes), then put him back in his stall and repeat several times.