I don't think that you are silly at all. Fear is fear whether it is founded or not. In the case of a horse, they CAN be dangerous so that is unfortunately stuck in your head. What needs to happen is you need to balance the possibility of something going wrong with the much much larger chance of everything going right and you having a lot of fun doing it.
When I was younger, I had no fear. I rode any horse, any time without thinking of what could go wrong. I even rode a bull once though not very long. Ha ha Now, I am older, I have seen some things. I have watched other people come off as well. When I think about it I can get myself nervous. Then I think "what caused all those problems to happen?" Usually, someone was screwing around, not paying attention or pushing a horse to do something that it was not ready to do. Avoid those possibilities and you have a far greater chance of complete safety.
When you think about how many people ever get hurt by horses to how many people ride every single day, the number is pretty small. Maybe you should practice emergency dismounts? Maybe that would help too.
Do you have someone that could help you by ponying the horse with you on? Maybe lunging the horse while you are on to gain confidence? Don't let anyone make you feel bad for feeling anxious about something. You need to move at the pace you are ready for and not a step faster. It will come when it comes. If it doesn't.... trail riding is still nice at a walk. :)