Barring health issues, you are not the boss of this horse. You need to establish yourself as 'the captain', and that goes back to groundwork, feeding, and basically EVERYTHING you do, and that WILL transfer into the saddle. Every moment with your horse, even in happy loving moments, you need to be establishing yourself as the boss. I never let my horses crowd me when I go to feed them - they stand patiently and wait for me to retreat when I am ready, I do groundwork randomly (just move their feet) if I feel even the slightest challenge, and if I have a new horse I am working with, I spend a good period of time letting the horse know that I am boss through ground, saddle, and basic interactions. I am kind, but firm - they move when I ask and come in when invited, and under saddle I spend a lot of time yielding the hindquarters if I am not getting respect (if he isn't going forward, make him yield his hindquarters a few times and then forward will be easier - not just turning the head, but YIELDING the hind end) - don't get off, but start PLANNING for 2 hour rides and just work on going forward even if it takes a long time (getting off is a release of pressure and, in the horse's mind, a 'win' - release of pressure = reward/horse establishing dominance). What do you do in the round pen? Are you establishing direction and speed? Once you do ground work in the round pen, do you get on and ride, establishing direction and speed under saddle? It helps to immediately reinforce the ground lesson with saddle work.
I bet you just need to really think about your interactions when not under saddle. A lot of times the way you relate to a horse at times like feeding and grooming sets the tone for the horse. Do you let him push past you for his food? Grab the hay out of your hands when you are carrying it in his stall? Does he turn his butt to you when you go to get him? Look at how he stands in the stall the next time you go to get him. If he isn't turning to face you with two eyes, you need to do some work. If he is, watch all your other interactions and make sure you are setting the tone.