I agree with you in that it doesn't destroy their coat in the long term. But short term, yes.
Just wanted to add, I don't clip my horse that I show, and probably never will, (except for mane,whiskers, hairy legs etc. lol) And he has the nicest coat ever! That lovely dark brown with lighter dapples. I would'nt clip him. I clip my arab cause he sweats horribly in winter and its impossible to cool him down.
If you cut the hair too short, yes, it is an off color
What is too short? Any clipping i've seen on bays/blacks has changed the colour a fair bit... Not cutting it short seems like it would go against the original purpose of clipping... Or maybe that's just me :]
But yep, i'm the biggest advocate of a nice coat acheived through proper nutrition, health, grooming and yes in winter appropriate rugging.