Another thing about horse riding is that it is more sustained. You are more likely to ride for two hours than you are to jog or swim for two hours.
Also all the incidental work that you have mentioned adds up. Carting full water buckets around, mucking out stables, if you don't have a tack room carrying a heavy saddle a decent distance. Walking up hills to find and catch your horse. Moving hay bales and chaff bags.
I also understand where the doctors are coming from. For most people "horse riding" is like a trail ride that riding schools run. Sitting on old plotters watching the trees go by. I don't think this would be very beneficial.
Still, putting in a little extra exercise at the horse is easier. Run to find him, move hay bales around for no reason, clean out your friends stall, take your horse for a brisk walk. My ultimate fitness tip involving your horse is to try and free lunge in a much to big area with a heavy rope.
I hate people who think horse riding isn't a sport. You can win, can't you?