I was galloping around in a field with two friends and one horse's nose collided into my ribs, and I clung on, but then when my pony slowed down, I finally let go an fell to the ground. The pony I leased at the time went running off into the woods, so one of my friends had to go get him, while the other stayed with me and was thinking of ways tto get me back to the barn. I ended up (thank heaven) with only bruised ribs, which lasted only a couple of weeks.
Anyway, after the incident, when I was better, I had a jumping lesson in a jumping field that was very uneven on the ground. I got really nervous, but i stayed strong, and finished my whole lesson.
I think that maybe it's a trust thing with your horse, and if you have to, take it slow for a few weeks. Just do mostly walking and a little trotting for a few days, then you can do more trotting, then when you get back your confidence, you can go back to your regular riding. I hope you feel better!