From all that has been said, it sounds like a physical issue to me. Have you had a chiropractor work on him? What about massage? That would be my first course of action. My warmblood could not pick up a correct lead when I first got him, not even when I free lunged him. I had the chiro. Out and his back was really screwed up, which was my feeling. He's had a couple adjustments and now he's doing ten times better.
Also, I would have his teeth checked. But not by a vet. Nothing against vets but they have no idea what they are doing inside the mouth. They spend one to two days learning about the mouth and that's it. An equine dentist is better. One thing that could be going on is your pony may not have a lot of contact on his molars. For a horse to know where his body is in relation to time and space, his molars must have contact. About 70% of the horse's perception happens in the TMJ and if the molars don't have enough contact the horse won't be able to tell what his body is doing. My warmblood has this issue. He had been over floated so he doesn't have the molar contact he should and the dentist said he isn't surprised that his canter seems "off." In my words, it's like he doesn't know what his body is doing sometimes, like how he carries himself or the shape of his body. That's because his TMJ is off, but thankfully he's only 7 so this will change. The dentist told me that in 6 months I should see a big change. Your pony may have also been over floated, he may have bit seats, his incissor angle might be completely screwed up....so many things could be going on. But not every equine dentist is created equal. The guy we use is certified by Spencer LaFlure and is also an instructor of Spencer's. Spencer has a completely different way of doing teeth, starting with the incissors first and not the molars. This has made massive changes in all of our horses and all of our friend's horses who Mike (the dentist) has floated. No one else will ever float our horse's teeth. I would highly suggest getting one of Spencer's people out to look at your horse.