Geez. Tell that to the retired Thoroughbreds lounging about the pastures on Kentucky retirement farms. Most horses are just as happy being pasture puffs as they would be under saddle, and honestly, most retired horses have deserved their rest! There is a time when a horse is simply too old to comfortably carry a rider, whether that time comes at 15 or 35.
You wouldn't ask your grandmother to start training for a marathon at 90. If she wants to go out and take a walk around the block, sure, why not? But it is taxing on a physically aged horse to be ridden, and it doesn't take a horrible injury to do it.
About the "taking the fun out" line, taking the fun out of life for who? You, or the horse?