Yup, I'm pregnant and still riding. All my doctors look at me like I have horns growing out of my head, but whatever.
I have been riding for 20 years, I have trained my 11 year old horse from when he was a yearling. I ride bareback, walk, jog and lope.
He's western pleasure trained, so his gaits are smooth as butter, so I'm not concerned about jarring motions. Also, my chances of falling off are pretty much the same as me getting into a car accident. I'm not concerned.
You really aren't going to harm your baby unless you get into a freak accident. And really, freak accidents happen in non-horse related areas, too.
A baby in the womb is not going to be harmed by just horseback riding. If you're a regular horseback rider. I wouldn't recommend a pregnant person to ride if they hadn't been regularly before that. Babies sit pretty safely in the womb, so in my non-doctor opinion, there's nothing to worry about unless you have an atypical pregnancy.