I agree with Marecare...he needs to be retrained, like as if he's not been trained at all.
I like handling a baby from the day he is born, but I DO NOT imprint them ONTO humans; doing this can cause such problems as you are experiencing, especially if you treat him more like a 'pet' than an animal who is going to outweigh you by 10 times when he's full grown. And put that together with a foal who is outgoing and extremely confident...you get a very disrespectful potentially dangerous horse!!! I would not handle this horse lightly...that he is not afraid to PICK SOMEONE UP, tells me he is extremely disrespectful, and you may need to have someone help you deal with him...yes, I am suggesting another trainer, but don't send him off, have someone come and help you with him.
Either way, he needs to be retrained, and retaught to behave around humans. His behavior at the fence is him telling you to "get out of my space" plain and simple. Start there, by making him move out of your space each time you go into the pen. Ie, don't halter him immediately, just because he is the 'first one there'. Make him wait until you're ready for him, and any 'nasty' behavior you get him to back out of your space once more. Only halter him when he comes up to you with ears foward and a friendly expression.
Round pen work would benefit this horse, but you need to be persistant in what you are asking him, and not just in there to 'chase him around'.
"The ideal horseman has the courage of a lion, the patience of a saint, and the hands of a woman..."