He might be acting ornery but it sounds to me like he might not have enough confidence and respect of it while in the saddle. If it were me, once up in the saddle, I would spend quite a bit of time rubbing your horse all over with the whip/crop. If he wants to move aware because he is scared of it, tighten up just one rein so he goes in a circle. Keep rubbing it all over him until he stops moving. If he's wanting to buck, you have to stop him before he does it and if you're a decent enough rider, you can sense when a horse is about to buck. Prevent him by already having one of your reins shorter than the other so his nose is already a 45 degree angle to the side or shorter. Once he stops moving, you can stop rubbing him with the whip or pat him while you're rubbing him.
I would keep doing this over and over until he accepts the whip. Once he's past this and you can move forward--walk, trot, canter--and when its needed to discipline him with the whip, rub him in that same spot with the whip.
Opportunity is missed by most people because it is dressed in overalls and looks like work. ~Thomas Edison