In my experience the horse usually lets you know or it becomes apparent due to health etc. I knew someone who had a 30 year old gelding they tried to retire because of his age. He went pretty stir crazy sitting in a pasture and was perfectly happy to resume his normal trail riding routine. I had a 28 year old mare that was still running barrels on occasion. She'd even win every now and again. She was ridden several times a week until her unexpected death and never slowed down. She put those younger horses in their place when it came to barrels and poles.
My thoroughbred gelding was retired from riding at 24 but that was mostly due to some health issues. (dropped fetlocks, tendon issues, suspected DSLD, major weight issues, etc etc) If he's sound and game for it I wouldn't be worried about it.