If you fear that being " big" and aggressive to make the horse back off will make it fearful of you, then you can move it off slowly. You must be glacially patient, and everso consistant.
You decide the boundary, and EVERY time he comes past the boundary, back him up . You "place" him where you want him. If you don't want to push him off with body language, take the leadrope , lift it up under his jaw and put backward energy into it to back him up, then you step away. If he steps forward, back him off again. You will have to do this many, many times. But he will learn that he will be backed up if he gets too close.
In concert with this, train him to stand NEAR you but at a distance and stay there while you do stuff. It's kind of like ground tying. when every you are leading him and you stop for a rest, back him off and make him stand a good 8 feet away from you. Just stand there, while you make a cell phone call, or pick daisies or adjust tack. Work a lot on the idea that he can be near you but not too close and when he stands near you, it's a peaceful place to be.
Lastly, whenever he approaches you in the field, you stop him before he gets close than about 4 feet to you. Then YOU take the last step to him so you can stroke his nose, halter him and such, but always be the one who does the actually approaching to the final connect, not him.