I've mostly always believed that each horse is different in their maturity and growth, so my answer would be "it depends." But I would say 4 is an okay time to start really riding, depending on the horse. But I've come to realize it isn't the opinion of many around me. I worked at a Thoroughbred breeding farm, so all of their answers were 2 years old. But at this place I'm boarding, although super relaxed, most of them probably have never owned a horse under 10 and that wasn't an already super-broke trail horse. They're all asking when I'll finally be riding my gelding who is probably an early 3 year old, not long 3 year old like we thought. He shot up 2 more inches in the past 6 months it seems! He is still quite immature, and I do not feel he is ready for actual riding. We go for walks, do ground work, all that fun stuff right now, which it seems he still needs a lot more of. Even my OTTB didn't really settle into a calm, mature disposition until he was 5 or so.
Anyway, excuse my rambling. Just wanting to discuss! For those who have started a lot of youngsters, when did you find they really settled into a mature mind suitable for work? Or simply, do you wait to back them until then or do you prefer to back them when they're younger?
Before I give you my opinion let me provide medical information which should be considered by everyone who is going to start actually riding a horse (since that's what you're asking....when is it good for working them under saddle).
I'll give development time for a filly of common riding sized horses (TB, QH, TWH, etc....) which have the shortest times. A horse's joint development matures from the ground up (with the exception of their neck joints). Now while it can vary slightly (by a matter of months), based on gender and breed size, it take 36 months for the leg joints alone to finishing developing from hoof to scapula (colts generally take about 6 months longer than fillies). It's 60 months before the back has finished developing (add 6 months for stallions...they are no longer colts at 48 months). The base of the neck is the last joint to finish, but since it has little to do with a horse carrying weight I won't get into that (but it does matter some with treatment which can impact the neck).
Now before I continue, let me point out that following the guidelines of the TB racing industry is about as intelligent as going back to surgical cleanliness procedural practices from 1000 years ago. e.g. it doesn't matter when your horse was born, the Jockey Club sets it's birthdate at 1 Jan. They race horses, running full tilt, as 2 year olds when they have not been out of them dams for 24 months. So forget those people and all the training age concepts (even for other breeds) that came out of that training philosophy.
Think of it in terms of children. Unlike dogs or cats where we like to say 1 year = 7 (which is not really accurate since it varies depending on their actual age), horses do actually age at about a 1 year = 3. So a 3 year old is like a 9 year old child in development. So ask yourself how much would you load on a 9 year old and make it carry for 10 miles. Of course to be fair we have to put the 9 year old on their hands and knees since that would be more accurate (horses are not structurally designed by nature to carry weight on their back) since we are better designed for carrying weight providing we are standing (but not on all fours...when we are more like a horse).
Personally, I've started training horses at 14 (1971). I've made mistakes that I learned from. I had old 1800's horse men who taught me things that false, but also told me to ignore things that many people believed (because of tradition), but they knew from experience were rubbish (like a horse needs to be shod...they proved beyond any doubt that it was rubbish using a white mare with white feet). Of course Gordon Naysmith shattered that myth around 1972 when he rode 10,000 miles, much over brutal terrain, while unshod :)).
Anyway. I'm not talking about training in general here. There is a great deal of training and even conditioning that can be done long before a horse ever feels the weight of a rider on it's back. However that is not what you asked about.
But I would say 4 is an okay time to start really riding, depending on the horse. But I've come to realize it isn't the opinion of many around me. I worked at a Thoroughbred breeding farm, so all of their answers were 2 years old.
I turn down people who want me to train 2 year old for riding or even 4 year olds unless it's over 56 months since I'll work them at being saddled and being mounted so I'm never going to be on their back for more than about 10-15 minutes per day while I wait for them to reach 60+ months (depending on size and gender.....stallions or larger breeds I add the extra months for). I'm certainly not going to be "riding" them.
Now all that being said. As I tell many people (including those who I've refused to start riding their horse too soon). "It's your horse so you can what you like. It's not my horse so I can't stop you. But if it were my horse it would be older before it started to be ridden". So all the people who read this and disagree...the same statement holds true (not my horse...not my problem). As for the development time for a horse's joints...well, doesn't matter what you choose to believe or not. Equine medical science is what I'm going by. That doesn't mean you can't ignore reality and go with tradition (even try to find someone to use 15th century medical practices to take out your appendix LOL)....or even what the Jockey Club says LMAO. Feel free to drink the Kool Aid if it makes you feel better, but don't make the mistake of thinking that you're doing right by your horse.