My vet believes if you don't use it, you loose it. In other words, he believes in keeping them active and using them so their muscles and condition don't deteriorate.
That said, I have a 30 yr old that I don't ride anymore, but that's because he never has a sound day. But if your guy is basically sound and feels good being ridden, I don't see why going out for an easy ride now and then would hurt him. It should be good for him, as long as you keep in mind he is an old boy and go easy with him.