I've started mine at two - ground driving - sometimes earlier just for fun because it is a way to interact with them as babies - but actually ground driving with a purpose? two - and I've hitched them to light easy entry as two yr olds to pull around the yard and at three, they had it down and I could start actually going somewhere, adding a passenger (besides myself), working on flat surfaces with some long slow distance (lots of walking for the first few months and slow gradual trotting) and some horses took to it so well we were already going to shows as three year olds. I'd say two years is a great place to start 'conservatively' but there are variables such as temperament, size, development, etc. Each is an individual.
Clippityclop is finally getting to spend some time in the saddle!