Yes, three is way too early. Three is the age you should start basic training, jumping is way down the road. At that young of age the horses legs haven't even fully fused together yet. By jumping him you're putting too much stress on them.
I personally would never even briefly consider jumping a horse before it was 4 years old, unless you're referring to some light free jumping. My Arabian mare didn't start over fences under saddle until she was 5 years old.
There is so much additional pressure and weight coming down on the joints and spine, it's absolutely crucial to wait until relative maturity to begin jumping with a rider.
My mare is 5 and I have only sone a few jumps up to 2;3" in her entire life time and that was only a few months ago, at three there knees and legs are not fully developed and they are mentally not ready either, I have not pushed my mare 2 hard, she has only been ridden on the flat from the time I broke her by myself until now, and she is 5 and a half, her flat work is amazing because that is all I ahve worked on, she has done a few jumps but deffinately not many