3 years is a good age to start putting weight on a horse.
One thing peope tend to forget is; if you let a horse stand in a flat pasture untill he's three, and suddenly start working him quite much, he'll probably break. Same thing if he's 5, possibly even a bigger risk for injuring him.
If the same horse has been in a good, natural pasture with friends, been led aroud, seen things climbed some hills, maybe jumped (without a rider), gotten used to a saddle, been worked in hand etc, the horse will probably be able to handle pretty much more than in the first example. They need to use their bodies even when they're young, to get strong. But not carry weight or overload their bodies.
And always start carefully and slowly, that way you'll get a good, sound horse. Train them as much mentally as physically, that way, when he's strong enough to carry you on 4 hour rides, he won't get scared and run off after 10 minutes when he encounter his very first mailbox ;)