It sounds like he's taking advantage of you and being lazy. Even if there were saddle fitting issues or pain, he would still move instead of dismissing you completely.
My first question - can you find someone else with experience to ride him? It should be very telling if they can get him to move.
My second question - Do you know how to apply the whip? When you ask him forward and he ignores you, you have to apply the whip lightly to his hind quarters. If he still refuses (which will most likely be accompanied with pinned ears) ask again immediately. Do this over and over until he goes forward. If he tries to go backwards or to the side, redirect his head and keep asking with whip, clucking, and nudging him until he goes forward. Chances are he will crow hop or buck, so just be prepared for that and KEEP ASKING if that's what he does. As soon as he moves forward, relax and let him walk out.
If anything, the next time you go to ride him tell yourself you are not stopping until he goes forward, not matter what it takes. You CANNOT feel discouraged or doubt yourself, you horse picks up on that energy right away. See if you can make him go on your own, and if not find someone who can ride him for you.