I know its a respect issue, but did I do the wrong thing the first time to promote this behaviour,
This is what you did wrong
He has always on the lunge turned in and charged at me, which I just gave up
Since you "gave up" when he tested you for your leadership.. he won. Now he knows he owns you. You aren't his leader and he doesn't have to listen to you. He's the leader and he's keeping you in line with this charging (this is from his perspective.. from what you're portrayed to him.)
So then he decides "oh hey.. I wonder if I can keep doing this to get out of work!" and then you get this problem:
But now he has began this behaviour when I ask him to do practically anything
And it's getting worse due to this problem:
I just flapped my arms to stop him from coming at me and then gave him a wack on the bum ( I wouldnt do this if I wasnt in danger)
He needed a bigger correction. One that made him feel like you were going to send him to hell and there he would die a slow painful death.
Okay not this severe, but seriously, rearing and teeth baring, and charging are BAD behaviors. And they will escalate, as you have experienced with your horse over time.
Don't be afraid to whack that horse with a crop until he stops rearing. Don't be afraid to smack the whip on the round when he bears his teeth to send him away from you.
If you are afraid, then you need to find a trainer to get your horse back to being submissive and respectful.. and then YOU need to get some lessons on how to handle horses.
Horses are dangerous, and if you don't educate yourself then you will get HURT.
Best of luck, please find someone to help you.