I agree that hunters is the best place to start. I do hunters and I like it better than jumpers. But that is just me. Everyone has their stance on what they like and as always in life--do what you love!
But first foundation foundation foundation. It is all fun and joy to run around a course jumping quickly, but if you don't have the right two point, striding, and balance foundation, you'll never make it far. Hunters will teach you that. It is like first learning to ride. I'm sure there are some exceptions out there, but from my 14+ years of horse experience, I've never seen a trainer teach a brand new beginner how to ride by sticking them in an English saddle. Usually they learn Western first before moving to English if the student so desires. It is the very same with jumping. Getting that solid foundation and learning hot to control your horse and move your body correctly is what is needed in order to speed things up and take the courses on a timed limit.
On the down side, it isn't doing what you want just yet. On the plus side though, it is still jumping. And it is still horses.
As a side note, if you own your own horse, know your horse. Some horses (I'm under the genre of jumping horses) do much better at jumpers. Some do much better at hunters. It is hard to turn a good jumper into a good hunter. It is possible to turn a hunter into a jumper, but know your horse!!
~He knows when you're happy
~He knows when you're comfortable
~And he always knows when you have carrots.