A horses behavior shows you what holes he has in his training or relationship with you. His lack of obedience and trust in the saddle is your clue to what needs repair. Try to be excited everytime you find a new hole in his training, because it is telling you exactly what bothers him. You can always ask folks here what to do and you will get lots of ideas to work with.
I agree with other posted suggestions and your own instincts. Spurs and whipping are not going to solve this problem, and fighting with him will create other issues. Once you are sure that a horse knows what to do, then you can punish him with a spank if he doesn't, but even then you need to pay close attention. My mare was very ornery one day and wouldn't go to part or our pasture. I was spanking her and it didn't help. I finally realized she was afraid of my neighbors llamas. I needed to desensitize her to them, not spank her.
If your horse resists leaving the barn, stay there but work him hard. If he will work hard with you in the saddle, back him, do circles, serpentines, and anything else you can do with him to keep his feet moving and make him WORK! If he resists in the saddle, get off and work him on the ground, remembering to stay out of kicking range. You don't have to be in the saddle to teach saddle skills.
Once he is puffing and ready for a break, ask him to move away from his preferred area. If he moves off in the direction you want, even if it is just a step, let him rest and get his wind a minute. Ask him again to move in the direction you want. If he refuses, WORK again till he is puffing. REPEAT, repeat, repeat.
After some time, and each horse is different in the amount of time it takes, he will connect moving away with REST and go further and further. Finally, he will be more eager to head out than to work so hard at the barn (or wherever he prefers to be).
This is making the right thing easy and the wrong thing hard. It is a great strategy, and not just for horses (teen aged boys come to mind).
Finally, think safety at all times. If what he is doing or what you are doing makes you uncomfortable, it probably isn't safe. If he isn't doing well at something, try to go down a level in what you ask, to something he can do successfully. Get both of you calm and successful, then try again, being very careful to reward every little try, and timing that reward in the INSTANT it happens. You will soon find the horse back on track and both of you much happier and safer!
Thank you so much for the ideas!! He will work with me in the saddle as long as we stay by the barn after that he throws his fit and tries to go back to the barn. I am going to start tonight with just staying by the barn where he is comfortable and doesn't fight and work him. Again thank you so much for helping not just telling me I shouldn't have him because I apparently have no idea what I am doing.