|And I have only jumped him about 3'9''|
If you have no dressage what-so-ever under you and your horse, you are doing an injustice to your horses education, physical wellbeing and to the sport.
You should not being doing 3'9" at all - when you have no strong fundamentals of dressage basics under neither of you.
Right now - you are probobly riding front to back and pulling on your horses face instead of lifting their ribs/back and focusing on the back end. You are more than likely, going around on a flat horse, heavy on the forehand, with no balance, no rhytm, no control.
Jumping is dressage, with speed bumps. You CANNOT be successful at jumping if you do not have dressage.
If GP riders spend most of their time doing dressage and jumping once a week -where their horses do minimally level 3 dressage - then so can you.
This is about the horse, not you. Do your horse some good and put proper training under the both of you.
I would far wrather see a rider going over a 2'0" fence, on a horse that is soft, supple, rounded, engaged, rhythmical - with their rider solid and fluid......than someone plowing over a 3'0" and bigger fence, with a flat, heavy, unrhythmical horse and a rider with unfunctional form.
You need to focus on dressage - no if, and's or but's about it.