^^But free jumping isn't the same as jumping a controlled course with a rider.
Anyway, I actually agree with everyone that has posted.
First, since you already have him and know him, give him a chance. Conformation is important but heart is more important. A horse who enjoys their job can overcome shortcomings in their conformation and be successful at a given discipline. How high have you jumped him? Did he seem to struggle with that height?
When I was younger and much dumber (dumber because I did it all wrong), I used to jump my QH, who had less than ideal conformation. He had no problem at all clearing 4'+ with a rider.
Also, he's only 4. Some stock horses will continue to grow in height until they are over 5 years old. He may yet even out some.
However, if it does become apparent that he truly can't do what you want to do, don't feel bad about finding him a good home that his talents are more suited to and buying a different horse that can take you where you want to go.