DancingArabian is right that you should have him respect you even if he isn't your horse. You should definitely take a lunge whip or some long stick like a carrot stick out in the pasture with you. They aren't used for beating, just to keep distance between yourself and the horse. Keep a radius between the two of you that is longer than his legs and, if you have to, you can tap him with part of the lunge whip to move him away. He will eventually learn that you won't have anything to do with him other than shooing him away and will probably stop bothering. It would probably be best to call and notify his owner with a heads up that you will be messing with her horse with a whip, just tell her what's up and I doubt she will mind.
If she DOES mind would it be possible to put his halter or an extra on him and tie him to another part of the fence while you put your horse in?