I am an open-minded person most of the time, but this is one area I am quite judgmental about.
In my personal opinion, a horse can be worked with from the time they are a foal, but I don't believe a horse should have anyone on its back until age three. I think at age two it is fine to saddle up and do some light ponying but nothing long or strenuous. Between three and four it is fine to ride but no jumping, speed events, or serious work. By age four you can start the horse's "career" so to speak, but I wouldn't consider a horse fully mature until age five.
Of course many people start horses much younger than I believe is right. I also don't believe a mare should be bred until age four or five.
My beliefs are based on many years of seeing how young Thoroughbreds snap their legs while galloping and how horses that are started young are almost expected to have joint problems by their early teens. People who start horses young also think a horse can be ready to retire at twenty.
Horses that are started later in life can reasonably be expected to be used into their late twenties or early thirties. My mare who was started late is twenty and is as sound as she was at ten. I don't have her on joint supplements or arthritis medicines and I don't expect her to need them until she is actually "old."
Of course I feel my horses' health is far more important than earning money or fame with them, and that's why I pour money into my horses and watch them poop it out their back ends.