My mare will do that sometimes during groundwork, but if I take a step toward her and yell no, that is the end of it, and she;ll go back to working quietly. But, I have the advantage of knowing that she is really a coward at heart. I'm not saying that is what you should do. I would keep driving him away from you with the halter and lead rope, then follow him from a safe distance and keep driving him off. Even when he shows that he wants to come to you, keep driving him off. Occasionally stop, turn your back on him and ask him to come to you, if he does not come to you, drive him off again. Keep repeating it until he comes to you willingly. This will actually accomplish two things; 1: it will show him that you are a dominant member of the herd by moving his feet and 2: it will defuse the aggression that he is showing toward you. When you do allow him to come to you, put the halter on him, reward him, then remove it and walk away for a while. Don't be surprised if he follows you back to the barn when you walk off. This basically teaches him that with you he gets to relax and away from you he has to work. One tip though, if you don't want him so close to you that you have to hit him with the halter and lead rope, carry your lounge whip with you, and use it as an extension of your arm. Also remember that you can change the direction that he is traveling without a halter and lead rope on him, simply by changing what side of him you are on. You can do this without ever leaving a walk.
All of that aside, you need to do a lot of ground work with this gelding. You have got to get him to respect you! And the sooner you do, the sooner his aggression toward you will stop.