It depends on your preference and the horse itself. My mare likes to nibble (she's 5, but she thinks she's still a baby!) and I don't punish her for it. My view on my personal horses: I try to bond with them as a horse might, I let her nibble or play with my clothes but she knows when I am angry and if I raise my voice to her she will stop. We have gone through the biting stage, and she won't bite me, so as long as she is respectful and knows where the line is drawn, I don't see a problem with her behavior. I see it as her being playful with me, bonding, whatever you want to call it.
My gelding is 28 and he loves to nibble/lick my hand. I have the same approach with him. He would never in a million years bite me or another person, he is just searching for treats. He is a W/T lesson horse, and the kids love him! Beginners are just forewarned not to give him small treats (and taught how to give them with a flattened palm) so he doesn't mistake their fingers for treats, but he usually finds the treats with his lips and tongue and pulls them in that way, he's never bitten a child- accidentally or purposefully.
I usually leave this behavior at the stall or grooming area, and they both know that if they are being tacked up or led, they should respect my personal space. I'm usually the one to initiate the behavior, and they respond. If they initiate the behavior and I don't feel its appropriate, they usually back off if I raise my voice and raise my arm. I just feel its a form of communication, and as long as they are gentle, there is no harm done.
However, if I was training someone else's horse or riding a lesson horse, something along those lines- I would draw a clearer line and make it more of a professional relationship with that horse. The owners can develop that sort of relationship on their own time, and I am not going to encourage what someone else may consider bad behavior. I will give them treats or praise, but I won't be as personable as I am with my own horses.
Since he is a baby, I would let him be a horse and a baby horse at that. The only time I would correct him or draw any sort of line is if he gets real pushy, bites, kicks, or otherwise tries to invade your space when he is clearly not welcome to. Because he is young, simply raising your voice to him should work with raising or waving your arm. If it doesn't, then carry a crop with you and just smack him (just hard enough to sting a little, I wouldn't try to hurt him as he's still figuring things out) in the chest [so he instinctively backs off]. If you are going to punish his bad behavior, you should reward good behavior, so maybe start halter training in the stall or something like that, so you have a base to reward once you establish the base to punish.