It's more than just speed and jump colors. Hunter courses tend to have a series of lines and, as noted earlier, part of the question is getting the correct number of strides in the line, being on the correct lead, etc.
Jumper courses are less about lines and more about turns and combinations. You might have one line in a course (or none). I swear half the challenge in jumpers is remembering the darn courses! You make time in jumpers as much by making your turns tighter as by kicking it into high gear speed-wise. And of course you can't just run at the jumps because if you knock a poll, that could be it as far as placing. Although there are different rules, Jumper classes also often have two parts, the regular course and the jump off. If you make it through the regular course clean, you go through a jump off, which consists of 4-6 jumps from the original course, but in an entirely different order (usually involving lots of tight turns.) The winner is the person who rides cleanest, then fastest.
Finally, Jumpers goes higher. I'm not sure what the cutoff is in hunters, but I think it is around 3.6-4 ft. Jumpers goes much higher, especially at the Grand Prix level.
Less interesting, there are some differences in tack too.
Which you want to do will depend on what you enjoy and what you and your horse excels at. Or do both. :)