If your pasture isn't spongy, he shouldn't have a problem. Just build him slowly, remembering that driving muscles are different than those used in longing -- especially when he's starting and stopping. My gelding, Mingus, loves to trot with his head in the air, but he is finding his balance and discovered the greater strength he has when using his topline to his advantage.
I tend to be conservative when it comes to my horses and I certainly overly cautious with conditioning, but I want him to continue enjoying driving, so I don't want to overface him. There's a great conditioning schedule on (as well as a lot of other great info) on minihorsecde.com
We have a great driving resource with Happs in Ethel, WA. Maureen Harckom is a great advocate of minis (or VSEs, as they are classified by ADS) and has numerous driving events throughout the year, including their big CDE next weekend (June 18-20). There are some excellent teachers up this way and many knowledgeable mini drivers (I'm a newbie comapred to them). We also have a group called NW Beachdrivers, a very loose organization of folks who get together each fall in Long Beach to camp and drive (or ride) our horses on the beach.