Three pages and everybody has missed the boat completely.
This is just another case of a horse that has been slapped at and pecked on and nagged at until he has gotten furious and it has turned him mean. This is not unusual. I used to get these spoiled dinks all of the time. Pecking at a horse is worse for the equine / human relationship than leaving that sucker alone in the pasture. Since you are the one that has interacted ineffectively with him the most, you are the one he has no respect for.
The others that have gotten after him harder and not just slapped around at him have gotten more respect out of him, so that tells me that he is not as far gone as he could be. If he was real far gone, he would attack everyone. HE IS VERY CLOSE TO ATTACKING YOU!
Next, let me clear up a lot of misconceptions spread by people that do not understand horses and what makes them tick:
You don't need to spend a lot of time with a horse. 10 minutes once or twice a week will do wonders if it is quality time that actually teaches a horse something.
Horses do not get bored and do not need attention. They need feed and water. I have horses that stand around and eat for months at a time and only get caught to be trimmed, shod, dewormed, etc until I need them to ride. They stick their heads in a halter, come to the tack-room, get saddled and ride off like they have been ridden every day. They NEVER lay a ear back or act unruly. THEY KNOW BETTER. THEY KNOW WHERE THEY STAND IN THE HUMAN / HORSE PECKING ORDER. They are perfectly happy with it.
Eating round bales 24/7 is not any part of your problem. Getting too fat is only a problem if he gets obese, has a cresty neck and shows any signs of having metabolic problems. Right now, I would say that is the least of your problems.
Taking horses for walks just tells me that some people still think they are 1000# big dogs. Horses do not need to be walked.
Spending time brushing and grooming is much less important than letting a horse know that you are in charge and he should fear for his life if he is not respectful.
Horses don't get 'naughty'. Kids get naughty -- not horses.
A disrespectful horse is not being disrespectful because someone is 'challenging him'. This handler should not be on the same level as the horse. That is what is wrong to begin with.
Horses like rule and consistency. WHEN they are handled or ridden, they want to know where they stand.
So, again, it is not the amount of time you spend. It is the quality and consistency of what you do during that time.
I will try to get back tomorrow and lay out more details.