Is there any way you can allocate an hour or so a day to him? Does he live on your property? If so I would suggest, if you want to keep him, you go out there, with a whip, and get after him when he tries to assert his authority. He can have NO say over you unless you let him. Catch him, get him a smallish area, and free lunge the little bugger. If you don't have a round pen, set up an area with tape and standards and make him work. If he is a pushy bugger, growl, use the whip to make yourself bigger, be IN CHARGE! If you're worried he will hurt you, pop a helmet on. Unfourtently horses are very large and very strong and theres always a chance things could go wrong. It's just the chance we take with our passion :o) Get out there for at least an hour a day and get that little sod LISTENING!
I've had my mare since she was a yearling and I did the majority of her training. She went through an 'ugly' stage at 4 and I just had to set her straight, QUICKLY. If you don't want to/can't ride him yet, at least work him. Once he is listening to you, get poles out, make it interesting!
Everyone is giving good advice but at the end of the day it is up to YOU.
Good luck :o)