In a nutshell - be firm, be persistant. Keep him MOVING, keep those feet moving and he can't go UP. Don't be afraid to smack him - carry a crop, it is not a "weapon" it is a basic extension of your hand, it can reach where you cannot physically without putting yourself out of position/balance. Work on him moving off your leg - forward AND sideways. Use your voice, don't be afraid to be FIRM.
There is no physical way you could "smack" him with your bare hand or a crop harder than another horse could kick him unless you literally let your temper get out of hand (which from your posts I don't really see happening).
I deal with youngsters all day long. The trainer we just sent our 2 y.o. Too was quite pleased to have a colt in his barn that "understood his place" but yet still has that "fire" in him. He likes to remind me that that will likely change while he is there - at home he is out in the gelding herd to help remember his manners, there he is solo - but I kindly reminded him that while I don't mind him having "attitude" at no point to I want to find out he has been allowed to become an "alpha" when in hand. The guy had not seen us work him before and allowed me the time to pull him out and work him with my daughter under foot like we did at home. When we were done he said that now he understood why the colt was the way he was - I expect obedience, I don't tolerate foolishness, I allow a bit of playfulness, but I expect respect and him to stay out of my space. He watched him stand while brushed by my 3 year old (with me lifting her up), she rubbed his legs (with me standing there), I lift his feet and she cleaned them, then he dropped his head for his hugs & kisses when I was done. Through that he was corrected verbally a few times and each time he moved back to where he was. Nothing that consistancy and time didnt teach. You'll mange it. Just don't be afraid to get after him when he needs it.