Sarahver just posted a great description about how to behave when a horse displays aggressive behavior in the thread under this section by Princecharlie. I will probably repeat some of the information here.
Even softies like me who work with horses all the time understand there is a time when you have to be aggressive enough that your horse believes you are going to kill him. This has to be done WITHOUT anger or spite (he is a horse and does not fully understand that when he kicks you that it might kill you). But there are a few behaviors from horses that should be met with zero tolerance. Kicking at you, biting you, stomping on you, etc. When these behaviors are directed intentionally at you, you must show extreme aggression toward the horse. This means loud noises, smacks, snapping with the whip, hitting with rope, anything that will not actually injure your horse but will scare him into thinking you could kill him if you wanted to.
Now if I were training your horse, I have to think that rule #1 while lunging is that the hind end never comes toward me. I would also feel that I had two safety zones: one where I was too close for him to kick me, and one where I was too far away. I would start out snubbed up to his nose, leading him. I would turn his nose into me and his hind end out of the circle. Then I would ask him to walk forward following me as I backed away. I would feed him a couple of inches of rope, move slightly toward a more normal lunging position, and if the hind end ever started toward me I would pull the head back in to me and use the lunge whip to get his hind end back out. I would work on getting a little farther away from his nose, but if he ever turned that butt or threatened me you would hear me yelling, snapping the lunge whip and whacking him on the butt with the whip. I would work up to a distance where I felt he could almost reach me with his kickers if he tried (but was still walking nicely with his butt angled out just a bit), and then I would scoot out to a distance where I knew he couldn't reach me. From this safe distance I would do the same thing: severely punish the hind end turning toward me, rewarding a gentle walk on the line. If he decided to back toward me, I would pull that head around to face me, using the leverage of the lunge line, or even running around to face his head and then regaining control. If he decided to charge me, I would first swing the lunge line back and forth hard so the buckle smacked him in the face, and if that didn't stop him I would have my arms up waving and smack him hard on the face with the lunge whip. He's only 3 and has not had problems with aggressiveness, so it seems to me that he should be easy to bluff and not very practiced at aggressive behavior. Even so, if you are afraid of him you need to find a real trainer (or just an experienced horse person) who can work on retraining him how to lunge.
You can ride and be around horses without lunging them. I'm amazed sometimes at how many horses people are riding around daily don't know how to lunge. BUT if your horse knows he can show aggressive behavior and have you flinch away from him, he is smart enough to know that he can do this when he is not lunging as well.
Be careful also that you make sure when he is not displaying any bad behavior that you let him go back to lunging normally. If your cues get confusing he may think he can turn to face you and avoid lunging. I would practice lunging at the walk nicely for at least several days before going any faster. If I found I couldn't control his head with a halter, I would lunge with a bit and bridle on.
OK, just a few suggestions in case you wanted some ideas for working with the horse yourself.