I agree with others, it depends. I don't take students under 7. My daughter just turned 6 and shows small fry w/t classes. She's been riding well since before she was born and on her own off the line for the last couple years, but she is in the barn with me 7 days a week. If she was just taking a once a week lesson I don't think she would be ready as early, being immersed in horses and sharing my love for them helps. I started the same way, my mom rode prego and I was riding on my own before I could walk.
Most young children don't have the attention span or eye/hand coordination required to truly "ride" in the sense of being on their own and controlling the horse. Riding with someone or on a line is totally different. I take lots of little ones along for rides, my 2 yr old nephew is hooked on horses already and I'm certain he only asks to come see me because I will take him out to the "Hooses".
Life is like a camera. Focus on what's important, Capture the good times, Develop from the negatives and if things don't work out, Take another shot.