Have you looked into our Natural Horsemanship subforum ( Natural Horsemanship) ? If not, I'd suggest you do, it has many valuable threads and advices - you might find something useful for yourself and your pony.
Biggest exercise would be friendly/trust execises. Use a training stick with a lash on the end of it, a bag or just a rope and get him confident with you throwing it over his back, neck, hindquarters & around the legs.
If he's nervous at any point retreat, either by walking away (leading him along behind) and throwing the rope ahead of you or smacking the ground ahead of you, or turning away from him for a moment. This takes the pressure off.
Then turn back and try again.
If he's super sensitive, this could take some time but don't get impatient or frustrated. Just smile and keep going. It took me 2 hours one day to get a friend's Arab gelding to the point where he would stand to have me throw the rope over him, and this was a horse who is "well-broke".
Just keep in mind: pressure motivates, it is the release that teaches.
Going out and just spending time with him, brushing him or taking him to grass are also good trust exercises.
Then when you know he's not afraid of you, you can start making him do things that will continue to build his confidence in you as a leader.
I love natural horsemanship and spending a lot of time doing groundwork. There are a lot of DVDs out there that may be helpful to you, altho not all natural horsemanship trainers are good, most of them are, but some are not.