It definitely depends on the physical and mental maturity of the horse. My current horse is 17.3hh and I love the fact he was allowed to maturity physically before being started as a 5 year old. So many injuries and disorders can be avoided by waiting until the crucial stages of growth.
There's a lot of talk right now about European Warmbloods being started at 3 and jumped heavily, when they break down in a few years they're imported to North America as 'dressage' horses. However, I have no first hand knowledge of this, just heard it in several conversations in different horsey circles.
I had an Appendix filly that was left to mature to 3.5 before she started under saddle work. She was easily to sell because she was left and not started at 2 and burnt out. The lady who purchased her obviously liked other characteristics about my girl, but she really appreciated getting a horse that didn't have a higher risk for joint issues, etc.
It's sad seeing a lot of horses just burnt out because the age they were broke (and other things). I have many friends with top show horses started at age 2, and now those horse are 7,8, and 6 and they have arthritus, and are just plain lame.