I've seen no deterimental effects whatsoever in horses turned out until 2-3 years old and then brought in for training. They may be bigger, but they also have more trainable minds then a weanling foal. I personally handle my foals from a young age, but I make the sessions as long as the horse I'm working with wants. The second they do what I'm asking, even a tiny bit, I stop, reward and end the session. You simply can't aim for perfection in a young horse, you have to be happy to compromise or you'll end up with a fight. An older horse is easier to push, easier to ask more out of because they're able to focus much easier.
For example, I tacked up my 2 year old Paint filly today just in an English saddle and a bridle. I had her in the roundpen and I asked her to move out at a walk. She's prone to cantering circles around me in excitement. Today, she walked a few laps one way, stopped when I asked, turned properly, and did several laps in the other direction. So I ended the session. I really don't care if it was only 10 minutes long, she did as I asked, and I ended it and rewarded her before she could think of spooking at a butterfly or acting silly.
Jynx was virtually untouched until this year as a 2 year old. She had human interaction but no training. And I'm having very few problems working with her, she's super intelligent, very affectionate and wants to please. So I don't see any harm being done with leaving her to be a young horse.