The developement rate of a horse is not really disputable. That hard running (not a dash across the pasture), which is what racing is, is damaging to developing joints is accepted and stated by the "non racing" equine medical community.
As for colt and filly races. They have their own races. There's even a triple crown for fillies, but it doesn't get the attention, because it's not as fast (usually...Ruffian comes to mind). But the results are the same. And you make my point by them pulling out winning colt so it can just be a stud. They could still race him during the season and breed him off season. Make a lot more money that way. But they know the risk of breakdown is significant and that's why they pull them out. If there was little risk of a breakdown they would keep racing them.
The numbers that hold up and stay sound are so small by comparison.
Of course I take issue with anyone who's working a horse at two. Whether it's racing or driving cattle. You train young horses, you don't work them. Training can be kept light and easy. The horse can be taught things as it grows. Working should start after it's finished growing. Do people in others areas work horses too young. Of course. But we're discussing the industry that has the most money and wastes the largest number of horses annually in the US. Which is TB racing.