I've not been pregnant yet, but hope to be working on that next spring (hubby and I will just keep practicing for now LOL). I am a family nurse practitioner and I provide prenatal care in a rural health clinic. One of my newly pregnant patients is an avid rider. So she and I researched this issue a few weeks ago for both of us.
Most recommendations are that you can keep riding as usual in the first trimester. Trotting or bumpy riding in general will not hurt the baby at all. Of course, if you are high risk for miscarriage (incompetant cervix, multiple miscarriages in the past, etc.) it would be best to talk to your "high risk" Ob/Gyn or midwife about this and probably not ride at all.
By second trimester its best to ride on fairly flat, even ground, no jumping, etc. Again, these things won't hurt the baby, its just that as you get bigger you may have a little more trouble balancing yourself, so ride carefully. First time pregnancies usually don't show until well into the second trimester, so you probably won't have balance concerns until the 4th or 5th month.
It gets tricky by third trimester, some sources say not to ride at all. I would probably leave that up to my patient's judgement and their individual circumstances. If you are comfy riding, not having a lot of balance trouble, your horse doesn't hate you for the added weight (LOL), then keep riding, but supervised and carefully. Do not go riding alone.
Remember, pregnancy is not an illness. It is a normal life process. Your body is made to protect that baby from all sorts of things - including the jarring of a trot. The only real problem from riding during pregnancy would be a major fall. Just take precautions to avoid falling as much as possible (the things I mentioned above). I have a feeling none of your husbands, partners, etc. have recommended that you quit riding in the car. A car accident would probably be more detrimental to your pregnancy than a fall from a horse would be. Just proceed with caution.
Good luck! Have healthy, happy babies and healthy pregnancies. And, no matter what the docs tell you, you're the one delivering the baby - they are merely there to catch!
I was normal...then I got a horse.