I'm with you. I fell and broke my back 10 years ago, and basically stopped riding.
But recently, I climbed back on. The first couple of rides, I was pretty scared, although I tried my best not to be because I didn't want it to transfer to my horse.
I've been riding her twice a week for three weeks. That's six times I've been back in the saddle now, and on the last ride we felt brave enough to canter!!! I did hold on to the horn, I'm not ashamed to admit, but we did it!!
Just take your time and your confidence will come back.