Give the little guy a break, it won’t hurt him. They didn’t start them till 5 or even 6 in the old days, and on some of the cattle stations I worked on here in Australia still do the same. Or did when I was doing that sort of work. These days personally I like to start them off the day they are weaned. I actually start riding them when I judge them to be big enough to carry me, so I guess somewhere around 2 but its more up to the size of them than any arbitrarily allotted age and the amount of riding they get is slow and gentile, If I do get them to go past a trot it’s not for long, and then in small doses. And these days, living thousand or so Klms away from the horses I’m training, and being a full time PhD student, I don’t have much more than two, three if I can get away with it, months a year to dedicate to the horses. And since I start them as weaners, that’s fine (once I have them going along OK, they get passed off to someone who can give them the work they need). But till they are 4 or so they only get ridden a few times a year, and then only after I guess them big enough. The rest of the time they go to a block of land and chill out being young horses. The time to think things through, I at least, have always found to benefit them. They will be a bit rusty as others have said, but I find a break tends to make them better mannered in the long run.
I’ll give you an example. In the current batch I have at the riding stage I have three there now, there was 4. The fourth was for my uncle’s granddaughter to ride. She was, how should I put it??, a rather unpleasant individual? Perhaps.... complete and utter a#@ hole?, yeah that’s probably the better one (the horse that is, not the granddaughter). Anyway I have trained a fair few horses, so I’m no newbie, and know a trick or two to deal with various horses, but she pushed me to my wits end, it was a fight every step of the way, I got her in a month, where I got the other three in a day or two. I recommended to my uncle, my cousin, and his stepdaughter (the horses prospective owner) that she belonged in the glue factory (I might have been exaggerating a bit but I’d had enough of her). The little girl had already gone and called her “Star Wishes” or something silly like that and fallen in love with her, and WAY over estimates, her ability to ride a horse and so wouldn’t listen to anything I said. Her stepdad (my cousin) was under pressure to keep the horse; so my uncle did what had to be done, and sold it. That horse would be a handful for a very experienced rider, it would have killed the girl, literally. Anyway, about a few months later I’d gone back up for another round of training and asked how the B@#$% was looking (unaware that she had been sold). He said a lady bought her and he warned her that she was a handful, but were both stunned after she walked up to my uncle to be caught, walked onto the horse trailer when he pointed at it and told her to get on it and floated to her new home like a little angle. Unfortunately for her new rider, it gave her the wrong impression of that horses capabilities and from what I hear she totally under estimated her and got, as we say in Australia, “thrown to the sh&* house” and was too scared to ever get back on her. But after all that the point is, giving them a break can be great for their attitude.