First of all, a horse should NEVER be afraid of a whip. If you believe that it should be, then I believe that you need to go back to square one in horse training.
A whip or crop is not a cure all. Nor is it a punishment. A horse should respect a whip and understand what it means, yes, but it should not feel as if it needs to run away or shy from it anytime that it is brought out.
Put the whip away. The first thing you need to teach this pony is respect. How are his ground manners? Can you lead him without worrying about him bolting, stepping on you, or stopping and refusing to move? If not, start there. If he DOES have great ground manners, you need to actually TEACH him how to lunge- not just start whacking him on the butt/legs until he starts moving. He most likely has no clue what he's doing. You normally lead him on a short rope and want him to follow you, and now you're giving him a super long line and you're hitting him or waving it futily behind him. He's probably just confused!
Have a friend(how about that neighbor of yours?) take his halter and lead him in circles around you. Use your voice to cue him. Have the neighbor stop and start him whenever you tell them to. He'll figure it out. Next, try to get him to move forewards without the neighbor guiding him. If he even takes a single step in the right direction- reward him. If he doesn't ask him again. If he still won't do it, try having the neighbor guide him for one lap then release. He should be fine. Using the 'ask, tell, demand' method is a good idea also, if he's just being stubborn AFTER you've shown him what to do for a few days. First, ask him to move forewards with a cluck or a word. If he doesn't listen, swing a rope a bit behind him, or snap your lunge whip (not on him, behind him!.) and use your voice. If he STILL won't do it, snap it closer or give him a LIGHT tap on the butt. He should move forewards. Remember, fear is not a good way to train anything, especially a horse.
I'd advise in reading up on horse handling before doing anything else with that pony.