I've gotta say I didn't read any other responses yet I just had to answer this right away.
I have a mare at my rescue, who's 28 years old right now and almost completely crippled. She was born on the Arabian show circuit worth $40,000 the day she was born. Her owners wanted her to be a halter horse, due to their intense training she was head shy and her neck was too sore to be touched by the time she was 2 years old. A kindly woman adopted her as a western pleasure horse - when her front end couldn't take it anymore the woman gave the horse to a teenage girl who was developing her dressage work. Within the next few years her hind end just couldn't keep up. She had at this point mostly gotten over being head shy and her neck was stiff but not bruised anymore, her front end was healing but arthritis was setting in. Her hind end was totaled, she crouched when she walked. We got her. She went onto an intensive joint repair diet and tons of stall rest. She healed enough to be comfortable in her field - but no more. She is now old and completely broken, but she is finally getting a chance to be a horse.
So my answer as to how far it should be taken: Let your horse be a horse before they're broken, they live about 30 years, you have a good 20 of solid riding in them, don't make the last 10 miserable.
Now I DO strongly believe that foals and yearlings should be handled extensively! They should be taken for walks everywhere and ponied from other horses and taught all basic ground work and in hand work that you can. But do not work for extensive amounts of time, they are babies, and do not add any weight or any pressure ont ehir growing developing baby bodies.
Your job now is to exercise the horse's brain not their body, their body isn't ready.