I'm 47 and just starting back after raising my kids. My daughter talked me into it actually....she's 15 and riding is her life. I'm scared of getting hurt too, but I had my first bad fall a few weeks ago and I was thrilled that I survived it with just bruises, more or less.
I was feeling bad about my age, though, since most of the riders I know are teenagers or younger. Then we went to the MO Women and Horses Expo and the winner of the equitation challenge was in her 70's!!!
There were LOTS of women riding who are older than me, or around my age. I felt a lot more positive about the whole thing after that.