I agree with both Kevinshorses and Spirithorse, to a degree each way.
Fear can certainly keep you safe. To that I say, if your knees are knocking and your teeth are rattling it's best to keep on the ground and get some confidence there. You being nervous, tense, and jittery will radiate right into your horse and make for a decidedly un-fun ride.
That being said, there are some times when you do need to give a gulp and just ride. My first horse was a chronic spook, and there were many times that I had to just suck it up and get on. The horse never did anything "bad," he was simply nervous and energetic by nature. If I was calm, he was better.
Something that truly helped my confidence was teaching that horse a solid one rein stop (incidentally, I started with lateral felxion on the ground, so the groundwork suggestion always holds
). By teaching it, I gained muscle memory in the cue, and confidence in my ability to teach the skill and to stop my horse from each gait. The training payed dividends when we were on a trail ride, and the horse shied from behind, bounding forward. Sheer habit and muscle memory kicked in for me, and one second later my horse was again calm, obediently flexing laterally to the bit. After that "incident," I quite literally felt like I could ride that spook basket anywhere, and always have enough control to stop him and dismount if I wanted/needed to.
If it's just nerves and "what if?" feelings, I would pony up and try to work through it. If it's a real "this is dangerous now, or will become so soon," kind of feeling, dismount and do groundwork for a while. That's good for both of you, too. Also, if you don't wear a helmet for every ride, do so. Even a quiet horse can be unpredictable, and wearing the thing and knowing that you are somewhat protected from injury there will help your confidence as well.
Good luck, and happy horsin' around!