We rescued a 21 year old horse almost three years ago. He is 24 now and still a great horse for riding, climbs the hills for hunting, packs, does whatever we want.
Just because a horse is over 20 does not make him an unusable horse, there are many many old timers on this forum who's owners still ride them.
Don't get rid of him just because the farrier told you he was over 20... When they get past a certain age, its hard to really decifer age because some horses teeth don't wear as much as others, younger horse can have "old"looking teeth depending on how they have been kept and fed.
If you are not happy with him, sell him, if not keep him.