LET GO OF THE REINS!!!
I know it's hard to do and you'll have to catch your horse, but think of your horses mouth. I've done this too even though I hated it. We were miles from the stable, but I did as much as I could to not let it pull on the horses mouth. He spooked, jumped sideways and off I went.
Also, don't brace for the fall. Tuck and roll but try to not tense up. When you try to brace with your arm as you hit, there's a lot of force when your hand hits the ground. That's where you'll break your wrist or arm. If you tense up, more likely you'll pull some muscles.