If he'll do it when being led, but not from the saddle, he's putting you on. First and foremost, do not address this problem until you have all day to fix it. You cannot let him get the best of you, otherwise you're only training him to misbehave. I would first ride him back and forth to and from the barn within his comfort zone. Make sure you get just as energetic a walk away from the barn and you get toward it. Once that's good to go, start increasing the zone taking him a little further each time. The game is, he cannot turn back toward the barn until you say so, and even then, he can only come back in a calm state. If he's fussing and figiting, he's going to be walking around out there until he gives up his tantrum.
If your horse starts to act up when walking away, here's a few tips. If he tries to spin around and run home, do not let him complete the turn. Use alternating reins to turn him back in the direction you want to go. If you let him face the barn, he's already won. You don't have to ask him to go forward at that point, but he must stay facing where you want. Once he calms down (gives up) then ask for forward again.
If he plants his feet and refuses to move forward, try the drunken sailor walk. Using alternating reins, take a few steps to one side, a few to the other, etc. He's wobble down the path zig zag, but he'll be moving forward.
Increase your chances of success by rewarding him with a return to home whenever he relaxes outside of his comfort zone. If you absolutely have to, you can decrease how far you go just get the desired affect, but try to go a little further each time, even if it's only a few steps. Do not pet, pamper and baby talk to him. He's being a butt. He doesn't need reassurance, he needs leadership. He will respect yours as long as you are consistent and clear with what you want. Good luck!
You just have to see your distance...you don't have to like it.