You can do plenty with a 3 year old that will help its future career. Slow trail rides of medium length introducing it to various things is a good start. Make sure the horse is comfortable with cars, random dogs, people and other horses on the trail. Work on balance under saddle, perfecting a side-pass and seat cues needed to go around obsticles, etc. 3 is a good age to focus on perfecting the basics that will help with your future distance trail riding.
Then as a 4 year old start the true conditioning with long slow rides, hill work, etc.