Firstly I think you should relax. there's no need to look at this as an all or nothing situation. It seems obvious that if you are distressed enough at the prospect of never riding your horse that you must love them, and the idea of riding them. It also seems however that you are a bit afraid to, at the moment.
So, you know what, that's totally OK, all of us have probably been afraid to ride a horse, or a particular horse, at one stage or another; I sure have. I guess Id encourage you to have a think about what it is exactly that you are afraid of. is it riding in genera?, is it riding one of your horses in particular? is it that horses are pretty big and strong? There are a lot of reasons that any reasonable person might be afraid of riding, so there's no need to feel bad about it.
I have ridden for a long time, and I have ridden some pretty nasty horses, and some really good ones too and I have started a reasonable number of horses off from scratch, and I have often felt a bit nervous about getting on them, particularly the young ones that might buck, so even with a fair bit of experience I still get nervous about it sometimes, I'm positive there's no reason you should feel bad about it if you feel afraid of riding either.
So keeping all that in mind, the reasonableness of being afraid, and the fact that riders with a bit of experience still get a little nervous, and I'm sure riders with a hell of a lot more experience than me do too, just think through why you might be afraid. Then think about what you can do to allay that fear. often tiny steps is the way to do it. If you are afraid to interact with a horse, start off small, just learn to be around it, to handle it on the ground. If you are afraid to ride it, just start off small, maybe just sit on it, take one or two steps at a time. build it up incrementally.
It means enough for you to have posted something on here about it, if you have a passion for it I'm sure you can overcome any fear you have.