I'd say take a step back and do some seriouse ground work for a month or two. If he's not even responding properly to haltering and leading, you probably need to go back to the basics.
For haltering. Talk to him when you're coming to get him. Try to get his attention as you approach, and when you go to put his halter on, slide it over his nose, buckle it, then give him a nice scratch (wherever his sweet spot happens to be. My mare loves her jaw scratched, he might like his poll rubbed, or his neck.) as a reward. When you're about to lead him away, turn towards him, not away from him. Pulling a 1,000 pound animal is going to be futile, but pressure almost always works. If he doesn't move out, press against his shoulder with your body, and use your leadline to direct him which way you want him to go (bring your rope under his neck and tug that way) while pressing. He should eventually take a step that way, and that's when you begin to move forewards. Always make sure you angle his face in the direction you want to go. A horse follows his nose.
When he stops while leading, use the ask, tell, demand technique. First, jiggle your rope a little bit to catch his attention and ask him to move forewards again. If he doesn't, do as with haltering- push into him, turning his head in the direction you want him to go. If he still won't move, demand that he does. Begin forcing him to circle (pivoting in his back legs, not walking around in a circle), giving him the option of walking forewards after each circle. Do this until he complies, walk him a few steps, then praise him and give him a scratch, or, if you use treats- a small tidbit. Continue forewards.
I would also focus on yeilding to pressure and desensatising. It sounds like he hasn't had much experience with people, and what he HAS had has all been negative. You need his full trust and attention.
Begin with running the leadline over him, repeating yourself wherever he is nervouse, and flicking it around him. Never touch him with it when flicking. Then try using a whip. He may act up more with this, but just continue your work. Flick it, twirl it, rub his legs and back with it.
Teach him to yeild to pressure at the poll, the bridge of his nose, the shoulder, the hip, and to back up. Teach him to flex his neck for you when you put pressure on him with the rope.
When he doesn't listen to you, and doesn't respect you, use the ask, tell, demand method again. Ask him to listen to you by trying to get his attention. If that doesn't work, begin to get firmer and firmer. It may take a swat or two (I would advise using your hand, not a whip, as this could make a bigger problem for you later on) to get him listening.
Once his ground manners are good, then begin his saddle training again. If you did his yielding exercises properly, he should be MUCH more responsive afterwards, to both your seat, legs, and hands.
I'm not sure what you're thinking lunging will do for him...at the moment I don't really see it doing any good, especially if neither of you are experienced. Just stick with the ground work.
Everyone in your life is meant to
be in your journey, but not all of
them are meant to stay till the end.