I know EXACTLY how you feel. I had a pony who used to take advantage of me. I used to be terrified to ride him, i used to make other people ride him for me, he would charge at me in the pasture, rear up on me, bite me, take off on me, try to lay down on me, try to rub me off on the fence. He didn't it because he knew that i wouldn't do anything about it. Thats your problem, your horse knows he scares you. He is taking advantage of you, completely. Don't let him. I was at the end of my rope with my pony, i didn't know what to do with him anymore, i couldn't sleep atnight i always cried because i was lost in what to do. What you need to do is get a trainer. Im not one for NH, but you need a NH trainer. Someone that will help build your confidence and teach you how to work out problems. Horses like that don't respond to smacks. and harsh punshment like other horses do. I find horses like "ours" respond to work. They would rather stay in the pasture and eat, the way of getting out of work is they scare you into giving up. You seriously need a trainer. I used to be so scared i would not let go of the horn, my trainer who was the sweetest girl. She really really helped me and him. I used to cry because i didn't want to walk on him. Horses sense that and they know they can get you off by scaring you. Horses like that are too smart for their own good. If you don't want to get rid of him get a trainer. Don't let him have power over you. You are the boss not him. And him scaring you, is him being the boss. Really, you don't have to sell him. You just need someone to help you. You cannot do this on your own. Im not insulting you, im just saying that you need help.
I know i made alot of assumptions in that, but it sounds just like what i went through with my pony Thunder.