It always depends on the horse. The most important thing is ground work. Work on it, A LOT. Do NOT think of getting on the horse until they have complete respect of you from the ground. You can do ground driving, leading, everything from the ground. Make sure the horse understands your space and theirs. Depending on how big yours is at 4 they can be quite a handful if they don't have good ground manners.
I got mine at 2 and he was 16 hands and hadn't been handled. I spent two years of doing nothing to ground work and bonding then rode when I he was a little over 4.
Be patience and get a solid foundation on him and you'll have a good horse. Don't rush anything. Get Dr. Beth Valentine's book, "Draft Horses an Owner's Manual" you can get it on Amazon
. It has everything you'll need to know about draft horses.
Good luck. And just remember - good solid ground work and respect.