Two is ok to start
a foal, as a natural progression of your ground training program, but still too young for any amount of time in the saddle, other than for simple, short (time-wise) exercises a couple times a week. No hard stuff, no jumping, no barrels, etc. Their growth plates are still soft and spending too much time with a load on their back can leave them with physical problems for life. Don't put a 200+ lb person on them for any amount of time at all.
Three is a good age for a breaking and little more riding, work in the arena, and maybe a short trail ride or two a week. They are ready for a little tougher exercise, maybe starting in their projected discipline. Still as a natural progression in the training program. Some horses mature a little faster than others, so it's still a matter of the individual horse's physiology. Unless you are a trained vet, you're not going to be able to tell, so it's best to go easy. However, there are a lot of 3 year-old race horses that seem to do fine, physically.
By four, pretty much all horses are physically mature enough for hard riding and tough duty.
Ranches often bring in horses from the range and break them in their fifth year. They break them pretty quickly and train them while working them, so they start right off working hard. Wouldn't work with a 2-3 year-old.
It's also good to keep in mind that horses generally are aged by birth year, not by birth date. So unless you know the actual date of a horse's birth, A "two year-old" could be anything from 13 months to 35 months old. When a rancher brought in horses from the range to be broke for ranch stock, If he took horses that were in their 5th year, he knew they were at least 3-5 years old and were physically mature enough for hard work.