As soon as possible.
I know some people let them keep their balls for a couple years to "fill out", but I always heard that if gelded early, the horse will actually grow taller than if they weren't.
According to AQHA, gelding a horse early has fewer medical complications, reduces the chance of having studdy behavior, the typical "cresty neck", and "Colts gelded prior to puberty often grow taller than if they were left intact until after puberty. The testosterone surge at puberty (18-24 months) triggers closure of the growth (epiphyseal) plates in the long bones of the legs, and the horse stops growing taller. The theory is that colts gelded at less than 1 year of age do not experience the pubertal testosterone surge and that allows more long bone growth and extra height."