OK, I didn't read everyone's responses, but my friend is just about to submit her book, America's Long Distance Challenge (revised edition) - which is a very popular and highly regarded book on endurance, and I had the privilege of reading the conditioning basics a few weeks ago. It covered this very question, so here's the summary:
Ages 5-7 are prime time to begin conditioning for endurance if you want a life-long endurance horse. As noted above, they are fully developed and can handle the stresses that endurance puts on their body.
Most people say that you can still start an endurance horse up to age 11, but she argues (from years of experience and research), that many horses 12 and up can be successfully started in endurance, especially if they have a working past. That makes the biggest difference.
Long, slow distance is important on older horses, but you'll find they condition a little differently and even hold it better.
And... if you want the awesome full-length explanation (as well as everything else you'll need to know about endurance), you'll need to buy her book. It's SUPER good! The old version is available on amazon and ebay, etc, but if you wait another month or so, her revised edition is coming out on e-book and in print :)