Just keep doing what you are doing. Be understanding and gentle but do not be afraid to be aggressive when need be. You must be firm, otherwise this horse will learn to walk all over you. There are many, many horses in this world that people insist were once abused but that is kind of a personal label for many of them. Handled rough is what I tend to describe them as.
I have a horse that would truley fit the abused/neglected category and believe it or not, he never feared humans. He has very little respect for them, men especially. He's not an agressive animal but he will fight if he feels the need. I personally saw him beaten with a 2x4. I saw the owners (teenaged boys) sit on their back deck and shoot him with a beebee gun. They thought it was funny when he bucked.
When I got him home he was over 200 lbs under weight and had a hole in his face about the size of the tip of my pinky that oozed puss and was covered with gnats. He had several loose teeth and this horrible scab on his withers from ill fitting tack. Biggest mistake I made? I forgave him. For EVERYTHING. I went to put the halter on and he ran... My though? Oh, he's scared. Nope, didn't want to be caught. But I walked away because he was scared. I taught him how to avoid being haltered. No halter, no work. Don't do that.
He would rear. Terrible rearer. I never got firm, he was abused. Can't be firm with an abused horse. Wrong! Be firm. I'm not saying be rough, just let him know there ARE boundaries or you are going to have issues. He's probably smart and if he is, he'll learn your buttons really fast.
As for the halter, treat it like a bridel when you are removing it. Don't pull it off, unhook the latch and let him pull out of it gently. Just like the bit, you let him drop it rather then banging his teeth.
I would get a vet out for a quick health check and as they said, age check but be gentle and firm. You'll do great.
"Be a best friend, tell the truth, and overuse I love you
Go to work, do your best, don't outsmart your common sense
Never let your prayin knees get lazy
And love like crazy"