Like Smarby said, do what you enjoy the most. You can always pull out the odd jump and go over that for variety if you really want to. Don't let other experiences always get you down.
In my lesson the other day, a girl in my class came off her horse coming up to a jump. Everyone waiting their turn said they then wanted to go last. When I volentered to go first, all of them stared at me like I was crazy. They asked my why would I go. I said 'cause it didn't happen to me'.
It is great to know the risks that jumping presents, but if you love jumping, don't let that experience pull you backwards. Of course, as you feel like you can take on the world in dressage, go for it!
I wish you best of luck!