Generally, the younger you geld a horse, the taller he becomes, the older you geld a horse (or if kept a stallion) the thicker, or more muscular he becomes. Stallions can, and sometimes do, continue to grow in their 5th, 6th, and 7th years. (I have a 6 y/o stallion myself and he is continuing to both get taller and more muscular. Although the height has slowed down quite a bit.
Unless you are going to be using your stallion to breed there is no reason to keep him one. A horse can be gelded as soon as both of his testicals drop. We like to do it at 3-6 months if possible. They are much easier to handle once gelded, you don't get any oops babies, and if you geld them before they know "what it's all about" then you have less of a chance of having a gelding that mounts everything he sees. (yes, a gelding can, mount a mare. She won't get pregnant, but diseases can be spread that way, along with harming your mare.)
There is a saying............A good stallion makes a GREAT gelding!
Get it done asap.