Also all the incidental work that you have mentioned adds up. Carting full water buckets around, mucking out stables, if you don't have a tack room carrying a heavy saddle a decent distance. Walking up hills to find and catch your horse. Moving hay bales and chaff bags.
I also understand where the doctors are coming from. For most people "horse riding" is like a trail ride that riding schools run. Sitting on old plotters watching the trees go by. I don't think this would be very beneficial.
Still, putting in a little extra exercise at the horse is easier. Run to find him, move hay bales around for no reason, clean out your friends stall, take your horse for a brisk walk. My ultimate fitness tip involving your horse is to try and free lunge in a much to big area with a heavy rope.
I hate people who think horse riding isn't a sport. You can win, can't you?