I have never started any of my young horses before the age of 3. For most breeds it's the amount of time they need to develop most of their joints (knees especially). If you had a warmblood (heavier bred horse) giving him/her more time to grow is more ideal.
There is still lots you can do in the mean time especially with a horse that has come off the race track.
3 years old is usually the ideal for most light breeds. Of the 3 horses I currently have one of them is a full draft so with her it would be a bit of a different case. I know I won't be getting on her until she is at least 3. Because they are so large, it really takes longer for them to fill in and their joint to close up more.
It really depends on the horse. Some lighter breeds also need more time to finish growing.