Well, its probably just a matter of changing your body language! Lunging correctly is harder than people think, it isn't just all standing in the center holding the line in one hand and eating a popsicle with the other!
I've observed many many people at shows and at home lunging improperly. The key is to keep your shoulders aimed at the rump of the horse... and that seems to be the hardest part for people!
This is what it should look like when you are lunging your horse from above:
The orange dot in the center represents a cone, or the middle of the circle.
The blue blob represents the human
The blue circle intersecting the human represents the path the human makes while lunging
The red line represents the lunge line
The green line represents the lunge whip
The green blob represents the horse.
Unfortunately my microsoft paint skills aren't spectacular, but you get the idea!
You arms are essentially in a driving stance. A 45 - 70 degree angle. The whole point of lunging a lazy horse that invades your space is to drive him out onto the black circle/track. You're aiming for a horse that is responsive, takes light contact with the line, and holds gradual bend towards the inside of the circle/track.
It is common for someone to try to walk backwards as the horse walks forwards, but that is not correct, and the horse will turn in or stop. With a horse like yours, you should constantly be walking forwards towards the horse, creating a "pressure buffer". You are driving him with your body, and to do that you need to be aggressive with your body language. This doesn't mean abuse or harm to the horse, but you need to have an active stance that responds when the horse moves in a negative way.
Start out by asking him to move off in either direction. As he takes off, start walking towards his rump. Your shoulders should constantly be perpendicular to the horses body, not ahead of the horse.... at the shoulder is iffy as well. Aim straight for that rump! Keep the lunge whip pointed at or just behind his rump. You do want to walk in about a 2-5 foot radius around the cone (and it helps to have a cone in the middle for reference!), always walking toward the horse. Think determination! Get his feet moving.... pop the whip, stomp your foot, get BIG. Use those driving aids, clucks, whatever you have chosen to use.
When he inevitably starts moving in to your circle, your first instinct will be to back away from him. Horses are big! As right as it may feel, this is teaching him that he is in control. Move the lunge whip towards his shoulder (like you said you do), and take one foot and literally quickly lunge towards him (yes, like the exercise move!). Stomp your foot forcibly as you do this and use the whip to poke him in the shoulder. He should and likely will move out of your space just a little bit. Reward this action by resuming lunging like normal. Getting lower to the ground than standing upright helps drive horses out for some reason, as well.
Repeat this every time he even thinks of creating slack in the lunge line. He will learn soon enough that he'll get an adverse reaction to it whenever he tries it. It may help to start out on a medium length lunge line so you can manage the length of it better and don't get tangled.
If he shows aggression, get huge, take two huge, stomping steps towards him, get lower, yell, crack your whip, anything to get him moving.
I don't condone abuse or beating horses, but a horse that comes dangerously close to you on the line can benefit (in certain situations) from one quick crack to the rump as long as you are out of range of their hooves.
Good luck! Don't let him intimidate you, and if after you try this you are still intimidated, think about getting a trainer to help/teach you how to lunge more effectively.