My dr. told me not to ride while pregnant, and seen how my dr. knows more about health, health risks and so on, I took her advice and didn't ride while I was carrying my son.
I feel that riding while pregnant is risk, not one that I'd be willing to take. It's a risk not only for you, but for your child as well, and why would you put your child in a potential harm's way? I know I wouldnt.
Yea, your horse may be the calmest EVER, but what if something were to happen, and you were to fall off, or kicked, or knocked over. To me it's seems like a silly risk, one that to me, is a no brainer.