I used to have very low confidence as well and thought I'd never jump again, but with a good trainer I pulled through. If you have the passion to ride you can definitely get to the level you dream of! It will take a lot of time and persistence, the right horse and trainer, but it's not unattainable.
If you can, get someone to film you every once in a while. I realized that often when a horse gets scared/bolts/bucks/etc, it feels much worse in the saddle (I used to get so scared), and then one day I saw myself in one of those events on film and I realized it was far from being as bad as I thought. That really helped me boost my confidence, because whenever I'd feel thrown about in the saddle, if I got scared, I'd remind myself that my fear makes me think its a thousand times worse than it actually is.
We all go through a period where our confidence is low, but a good rider is one that can face those fears and slowly (doesn't matter how long it takes) improve. You stayed in the saddle to start with! Be proud of that! :) Think positive and look at the good things, the stuff you can do, not the other way around.
Good luck! :)