He's becoming barn and buddy sour, and you are encouraging it by allowing him to stop and greet other horses, sniff manure, grab bites to eat, stopping on his own whim. My solution, at least right now, while it's in the early stages, is to move faster. Trot him out good and hard. Be energetic. If he stops, ask for forward again, then demand it. As others have said, use the crop like you mean it. No nagging. Don't use it unless you mean it, and then MEAN it. If he backs....let him. Make him back faster. When he wants to stop, back him some more. Then ask for forward again. It may take a while, or it may take one time. But as long as he isn't offering to rear, keep him backing. Even if you back all the way home. If that happens...Back him past home and then ask for forward. Don't stop until you've gotten satisfactory forward motion.
My leased horse has a tendency to want to sull out toward the barn when we're passing it on rides around the farm. I've been trotting her energetically past it every ride, and only stopping her once she's trotted past it with her body still between the reins. She's improved greatly, but she's another one that would take advantage of a green rider to become barn sour. I won't let it happen.
Problem is that the horse is smart enough and opportunistic enough to tell that you aren't the boss in your relationship, and he is taking advantage of you. It will become habit, and it will get worse. Get serious about it before you need a professional to sort him out.