I figure it this way, my horses, whether baby, or adult, have approximately 20 hours every day to 'be a horse'...a couple of 1\2 hour sessions each day is not going to harm him or interrupt him growing up and knowing how to act like a horse.
I want a youngster to learn how to do the basics as soon as I can, because it makes training undersaddle so much smoother later on; quite literally most of the horses I've trained from the time they were little ones, the first ride was like we had ridden many times before. They learn all they need to know in the first few years of life, that going from 'ground to saddle' is not difficult at all for them.
I don't think you are doing anything wrong with trying to teach your baby the basics this early on...as long as your actual teaching sessions are brief, spend all the time you want with your horse. Even going out and sitting in his paddock can have its benefits, especially if he is still leary of you. Chances are, the reason he IS leary is because he hasn't had the handling early on. Just be patient and keep doing what you are doing with him.
"The ideal horseman has the courage of a lion, the patience of a saint, and the hands of a woman..."