Sometimes I go through a week or so of just giving him tough love, pushing past him and making him do what I want, not being nice at all. But it never lasts for long :( often because I feel sorry for him. I need to really try and make it a long-term thing, with the things that you mentioned.
Yikes. This is probably 90% of the problem, right there. What this horse knows now is that sometimes he can get away with it, sometimes he can't, but if he can't get away with it NOW it is only a matter of time. All he has to do is keep trying and eventually, you're going to let him.
That's a bad thing to teach the horse, but unfortunately, you seem to have done it. You can undo it, but it's going to take a lot longer to undo than it did to do it in the first place. But it CAN be done.
I doubt you need a big arsenal of tools to do this. I wouldn't try loading up on all of the approaches that have been suggested here. What you need to do NOW is to decide - for yourself, and for GOOD - that you are NOT going to put up with this behavior AT ALL, EVER. And then you need to mentally prepare yourself for a long, extended battle.
You have to treat horses just like you treat kids. Decide what behavior is OK and what is not, communicate that in some way that the horse (or kid) is capable of understanding, and then keep those rules 100%, all the time. No exceptions, no feeling sorry for the poor thing, no bending the rules just this once.
Most of the suggestions in this threat have to do with the "communicate that in some way that the horse can understand". The solution is not to do all of these. The solution is to find one that works - and it sounds like you've already got one, with the crop - and STICK TO THAT FIRMLY.
That's not to say that it will always, forever, be a bad idea to bend the rules once in a while, but you're not even close to that point. All you do when you bend the rules, or break them for yourself, or get tired of fighting and stop, or change the rules...all you're doing now is confusing the horse.
A confused horse is an unhappy horse. And a confused child is an unhappy one, too. Both of them, they like to have structure and know who's in charge and what the rules are. Both of them, even so, will test once in a while to make sure that you're still in charge and that the rules are still the same. If you want a happy horse (kid) you need to make sure that you're delivering the same exact message every. single. time.
It is exhausting, especially when you have a bright, curious, pushy horse. I know this because I do too. But you have to trust me - and I know that other people here know this as well, including some of the people who have already weighed in...I'm sure they're thinking "Oh, no, so you've been giving intermittent rewards? No wonder he's acting out." - IF you can be 100% consistent with your boy, and you can enforce the boundaries 100% of the time, and you are communicating it in a way he can understand, THEN you are going to have one great horse, and a horse you can feel very comfortable with, and a horse that will be able to relax and love you back as much as you love him.
I will bet, since you say that your horse behaves better under saddle than on the ground, that you are a LOT more consistent about your aids and corrections when you're riding than when you are handling the horse on the ground. That, alone, will explain this whole mystifying phenomenon.
As I said, I'm speaking from experience on this one. My horse still mounts the occasional challenge. He has to. He does it under pretty predictable circumstances now, though, like if he's been stalled up a little too long, or hasn't been ridden or worked intensively in too long. He also does it when it's very windy, and when the weather is changing a lot. My job is to pay attention to the circumstances, anticipate the behavior, and be ready with my corrections when he crosses the line (because I already know he will). The challenges go on for a lot less time, and they are more "token" challenges, and so they're less annoying and time-consuming to deal with...as long as I'm ready with that correction and nail him (figuratively) as soon as he crosses the line. SO it's not going to make your boy a perfectly behaved horse, but it will be a lot more manageable.