I agree with having a chiro. out and your vet if that doesn't help. His attitude could very well be due to pain.
If you rule out pain, then it seems he's just testing your leadership. Sometimes we do things to horses that they hate, like micromanaging them, giving them mixed cues, being too aggressive, being unfair, etc. WITHOUT realizing it. This horse, to me, sounds like the type of horse (if he's NOT in pain) who uses his size to intimidate. We can't be stronger then the horse, but we CAN be smarter in our strategies.
This all starts right when you go to get him from the stall/pasture. Good horsemanship doesn't start when you put the halter on or start riding. It starts from the moment your horse sees you. Does he look at you with his ears full forward, eager to come out? Or does he pin his ears or ignore you completely? These are the things you need to pay attention to. Respect his thresholds as you approach him. If he turns even 1 ear away from you STOP. Wait until he pricks it forward then approach. If he even turns his head away from you stop. If he's in his stall, DO NOT go in. Open the door some and cause him to come to you. If he's ignoring you, use a rope to lightly flick him on the leg or hindquarters. When he acknowledges you, stop and smile. The idea here is to get him CURIOUS about you. Then you have him metally engaged.
Horses like this are usually very sensitive to suggestions. If not, they have been dulled. They're very observant and can tell your weak spots. When you ask him to lunge, try to be very soft but very clear about what you want. See how LITTLE it takes to do something. If at any time he pins his ears at you, charges, etc BACK HIM UP until the look on his face changes! You can do this by walking towards him, waving the lunge line and whip back and forth in front of you. When his expression softens, stop and let him wait a loooong time. Look for his eyes blinking, licking and chewing, etc. Invite him in to you and rub him for awhile. Then continue. When he acts that way he's saying "I'm alpha, you're just a wimp, and you aren't worthy of my attention!" So now you have to out smart the smarty! This kind of horse is constantly plotting so you need to be able to redirect him in a nano-second when he plays up.
As for bucking while under saddle, I would ditch the spurs and whips. This kind of horse will retaliate against forceful things like that and the bucking could escalate, or worse. Again, if you can 100% rule out pain, saddle fit, etc. then here is what I would do. The bucking under saddle is just a unwillingness to go forward. You know where that starts? ON THE GROUND. If he doesn't go forward willingly and softly on the ground, there is a high probability that he isn't going to want to go forward under saddle as well. So get it good on the ground before you mount up. If that means not riding for a couple sessions, then that's ok. Take the time to get things right. While in the saddle, again, see how LITTLE it takes to get him to go and to slow. If he bucks at any time, ride it out and keep asking but don't get aggressive. You need to out persist him, not punish him. The more forceful you get, the more forceful he will get. When he does as you please release all pressure and praise the heck out of him! Give him a rest to let him know without any doubt that THAT was what you wanted.