Make the effort to catch the pony so it can be moved, or make sure he's in a secure enough area he's not going to hurt himself trying to get out. Those are your options. The pony might be buddy sour. Use grain to catch the pony if needed. Otherwise you're going to be focused on whether the pony is going to get out or get hurt trying to get out and not focused on your horse when you have him out.
Your boy needs to learn to put his head down so you aren't on your tippy toes. Practice putting pressure on the lead, when he gives his head, give lots of rewards (scratches, treat whatever it looks like he responds to best). You can give him a cue too, like head down, or down or whatever. Keep practicing but in small amounts of time so he doesn't get bored with it.
Unless it weighs a ton... it's just a horse. Draft horse motto.