If your doctor is supportive of you continuing activity, I suggest you get your hubby to go with you to your next check up and have the doctor tell him it is good for you to keep up the activities your body is used to. With the first, I rode until I was literally too big and off balance to get into the saddle, about 8 1/2 months along. My friends joked that I would go into labor on the horse. I was actually cleaning stalls when I went into labor with my first. It was probably a good thing to speed it along, better than walking, IMO.
I didn't do as much with the second, because I had a c-section with the first, and was constantly uncomfortable from about the 3rd month on, because of the pressure being put on the scar tissue. But I still rode and worked in the barn until my body told me enough was enough.
I think it's sweet that he is concerned, but I think he is taking it too far, and likely it will take someone like your doctor to ease his fears. And even then it may not.
"Keep a leg on each side and your mind in the middle"