Remember that the only thing different to the horse is adding the weight of the rider and tack.
They gallop, trot, run circles, change leads, etc. on their own. That is if they are allowed to grow up as horses--in a big pasture with other horses.
Starting or riding an unfit horse-regardless of age--is a problem which greatly increases the risks of injury.
Most performance quarter horses are started at 18 to 20 months to get them ready for the futurities. Most "trainers" who start them do not manage the increased risks of starting them that young well. And yes there are increased risks but most of them can be managed to minimise the risk.
Remember that these horses are managed from the day that they are born to be a performance horse--proper care, nutrition, environment and knowledge.
That said for a horse that has not had all of the above then 3 sounds like a good number.
The key to all of it is the proper managment of the risks.