The other side of the fence! No shame in keeping yourself safe; in fact it shows you have savvy! A handy/carrot stick is good for extending your arm reach to deal with him; otherwise, a long enough rope (12' is great!) with a leather popper on the end will do.
Don't get mean or mad, & don't pop him any harder than another horse would, ever. Match his energy plus 4 ounces. If you display temper to him, he's won the game!
Back off your energy immediately after the pop, & back off walking
to reward his slightest try, to see if you can draw
him back to you (he turns back up to fence). If he comes back up to fence nicely, reward him with rub, treat, or just standing back & saying good boy if his body language says to.
Another way is to offer a treat, then offer treats sporadically (not each time), till he's always wondering if you have one, coming to you respectfully to see. That's not spoiling him, it's Friendly; only spoiling if it's done every time, & thus becomes a bribe that he's trained you to have!