Sadly, what you have been doing in persisting until he will go a little ways and then turning him home and ending ride, is just reinforcing this, as he doesn't equate "oh I took two steps and get to go home, so this isn't all bad" as much as he does "I won, hahahahaha."
He must be made to go ahead and work, and walk on where you want him to go, instead of cutting ride short, because he did a few steps more or less.
And when he starts the backing? Turn him and make him back into the direction you wanted to go in the first place.
But don't give in to him, or trust me, he WILL begin to not want to go any direction at all. That is coming. And probably soon.
Horses do not think "9 out of 10 times when I have had to be ridden, I didn't get ridden, just caught and petted and turned loose. So I imagine this is another one of those 9 times."
Instead they bank on averages, and bank low at that. "1 time out of the last 2000 times this fool was out here, I was ridden. Well, not going to be 2nd time for sure."
When this horse balks, either be prepared to sit it out, keeping horse facing direction you want to go, and sit quietly, no talking or babying him. And don't drink a lot either before hand. Also no fooling around with friends, cell. Your mind needs to be clear and fairly empty, not worried about anything. Just sit there. Every 10 minutes or so, give horse cue to move off, if horse takes a step do not praise or pat, just ride. Patting/praising breaks horses momentum, and that you do not want.
If horse tries to spin around, turn back to correct direction and go back to sitting. This may take 30 minutes or even more. That is why it is important for horse to feel like the sitting is your idea, not his.
Also, be sure there is no other area he is challenging your leadership, feeding/leading or what have you too.
But don't give in if he only does a little of what you are wanting. You are teaching him to keep at this when you do that.
Horses make me a better person.