Heading back to barn at a point horse decides "here and no further" only will make this worse, as horse will start shutting down at entrance to barn IF you can get it that far, that is.
Do not drink anything, go to bathroom before you mount up.
Go and when horse shuts down, make sure it is facing the way YOU want to ride, and then just sit there. Period. No urging, no battling, no clucking/kicking/kissing. Do NOTHING. And don't talk.
Horse may try to turn to barn and calmly and quietly, turn horse back to where you are wanting to go. And then sit.
IF horse makes a step in that direction, merely let it go that way. Do NOT praise, pat or talk. Just ride.
The very instant it shuts down again? Sit there.
As long as horse is facing the way you are wanting to go, do nothing. If horse tries to turn around, put it back facing way you want, and then be quiet and sit.
Do not battle at all, nor urge horse on.
Taking away the fight/battle, and remaining calm, will work wonders here. Horse won't have anything to throw a fit over, or refuse.
And again, no talking. No patting or praising at any point.
When horse gets bored and moves on out, be prepared for it to stall again, and just sit if it does.
It will get tired of this, and decide riding isn't that bad after all.
You can also, if you can mount easily, take a ration of grain out and at the furthermost point of your ride, you can grain horse....that will break the "oh I am going to eat at barn" thought.
However, you might be well served to have a trainer put a finish on this horse too, if you aren't that experienced, as two greenies together usually don't work well.
But again. Taking a horse back to the barn to "work him" or letting a horse decide where it wants to stop, is a bad thing to get in habit of. In the former, the horse doesn't see it is being worked as punishment, it sees it as "I am back at barn"....and in the latter, horse will simply learn it doesn't have to go anywhere at all.
Horses make me a better person.