I don't necessarily wait to geld, but I'm not in a great big hurry either, unless the colt is a snot or I can tell from birth that he's not good enough to be a stallion. I do wait until the first winter, at least, so that I'm not fighting flies on top of sore male parts.
I have gelded very late, 5 & 7 years, twice and had no issues. Both were excellent, quiet boys as stallions, they were both excellent, quiet boys as geldings. If I'm not using them as stallions, for any reason, I go ahead and cut. They can go out on pasture with the whole herd then and it keeps them very happy. I am a firm believer that a great stallion will make a phenomenal gelding and keep very few horses intact.