Agree with Ottakee that the very first thing is to check his saddle... back, neck, teeth, etc. Rule out pain before treating it as a training issue. Balance International have some good info on their website if you need it.
On that note, I would not be using a stirrup to hoist yourself up from the ground - at least while training & as a rule. Even a perfectly balanced & braced horse with a perfectly fitting saddle & perfect rider will feel some level of discomfort as he's pulled(often dragged) off balance if only for a second, especially if he's too big for you to spring up on. Therefore I recommend using a mounting block to make it as easy as possible. Ensuring he's trained to allow you to mount from either side will also even out the amount of discomfort he feels, especially when you come to mount from the ground.
It's either the actual mounting or your riding that he's got a problem with, so I would also want to rule out problem riding - ensure you're sitting balanced & riding him well - get someone experienced to evalate this who can tell whether he's uncomfortable & unhappy with you up there.
Below is a post I wrote previously on this issue...
Basically the trick is, work towards what you want in baby steps, persevere at whatever stage while he's moving away and reinforce his smallest 'tries' along the way. Make the 'Right' things easy and Good and the 'Wrong' things difficult and unpleasant. It will be easier & more effective - & fun for you both - if you add positive reinforcement(something Good) such as a treat, scratch on the withers, etc to the negative reinforcement(taking away a Bad Thing) of removing pressure. This is effectively the principle of desensitising & training a horse for anything.
Starting from scratch, maneuver him into position beside the mounting block and then stand on it & get into position. I expect he's already started moving by now? Just keep putting him back there & repeat it until he decides to stand. *the instant* he does, reinforce him positively(reward) with a treat or scratch & negatively (remove pressure) by getting down.
Once he's good about standing at the mounting block & allowing you to be there, for a while, just standing to begin with, then fussing with stirrups, etc, then the next step can be foot in the stirrup. Or even an intermediate step of standing facing him & leaning on the saddle as if you're about to pull yourself up. Hold the reins short but not tight so long as he's standing, but when he moves, tighten them a little to add a bit more discomfort to his 'Wrong' behaviour. The instant he stands still, drop the pressure - reins and weight/intent - to reinforce the behaviour. Repeat this until you're putting on and taking off the pressure without the horse moving.
While you're working towards training him to be mounted alone, it will help if you've got a helper for this bit, to minimise the difficulty and amount of time you're hanging off the side! Don't try to stop him from moving away, but with your helper, try to ensure you stay with him, foot in that stirrup, so his behaviour doesn't work for him. The instant he stops, reinforce that. Repeat the process until he stands to allow your foot in the stirrup. Remember to repeat(reinforce) this successful step many times without going any further. This is the step that is hardest for you & the horse, so make sure it's really strong & well ingrained.
It's also a good idea to increase the time you stand there(horse's patience) in small increments AFTER you've got him doing it for a short time, as with the above. When the horse has mastered the above, start keeping your foot in the stirrups for longer before removing. If he gets antsey, just hold a little pressure on the reins & go with it until he stops, again reinforcing immediately.
The next step would be half mounting. You might start this by bouncing up & down, between the ground & standing in the stirrup, or you might start by standing & staying up in the stirrup until the horse is still. Depends on how safe you feel at this point. Either way, negatively reinforce immediately he stands. Repeat this exercise as for the above before putting a leg over & doing the same.
It will also be beneficial to work on this in very short sessions rather than trying to get it happening all at once. You can do about as many 5 minute sessions each day as you like, so long as you break them up with some stress free relaxation or games in between. Even if this also is only a few minutes.
After the pony has learned all this, it will also be to your advantage to continue these exercises regularly for a while, without necessarily getting on. Then do them less often, until it is just occasionally and when you need to. Once he learns that he can and that it's easier to stand and brace himself for you, he will choose to do this, so long as what follows isn't too hard or unpleasant for him.
I realise this all sounds rather tedious & long winded, but whatever method you use, there is no quick fix to good training. Also, while initially it will indeed be tedious, as you progress & he gets the idea, it will get quicker & easier to get what you want from him, and you will find that handling his fears or problems in this way will show him you're considerate & fair and will enhance your whole relationship. These lessons will also rub off on other issues, making them much easier & quicker to deal with.
Best wishes & have fun!