Everyone takes a good spill every now and then. By getting scared, it's probably affecting him. Since you have a strong bond with him, you need to act as the leader. If your scared, then he might think he has a reason to be too - Just like when you were riding. The first horse spooked and he might have thought, "oh no, she's afraid and took off, so it really must be something to run away from!"
The world is best viewed through the ears of a horse.