It really isn't the walking and trotting that does them in, it is the weight they have to carry and the way the must compensate to balance themselves. IMO it is way to early to start a horse, even if they seem ready or if it is common for that breed. There are so many things you can do on the ground from the moment they are born, that even if you only have the one horse to break, you could spend years just perfecting and preparing them. Horses that are rushed into the saddle with gaps in even basic groundwork can be quite the hassle once you start asking different things of them. From what I've observed, usually once people start riding a very young horse, and if it doesn't go entirely bad, even if they planned on giving that horse a break, they keep with it. If you are going to start a horse young, it is best to get the basics through their head, then give them a long break. Once you pick them back up at three or four, you can 'restart' them to see what they remember, but if you did good work with them, it shouldn't take very long.
I've seen young horses rode into the ground until they are sweating and shaking and then asked for more. Not much of a good impression to be left on a horse if you want to quick drill everything into their heads. Good things take time, and the less experience you have, and depending on the time you are willing to commit, a year's worth of groundwork might not be a bad idea. I was started on a green three year old, and after getting bucked off a few times, he got about a year and a half of ground work because I was too afraid to ride him. By the time I built up the courage to get back on he was a very good horse barring the saddle fitting issues and bad teeth. Once those issues were resolved he is just a stellar horse who needs more miles. I've rambled quite a bit, but the point is take the time it takes! Good luck.