In my opinion, would not saying you should consider the horse first than the color be better than saying you shouldn't consider color at all. Honestly, if I found two horses that looked the same and one was bay and the other chestnut... I'd take the chestnut. Why? I have a fondness for chestnuts. Just like I'd take a palomino over a buckskin, a chocolate over a silver bay and an Appy spotted over a pinto. I don't think any less of the other colors, its just my personal preference. No different then wanting a neon green saddle pad over a dark blue one. They both get the job done.
Now I understand people do seem to make silly choices on the basis of color alone, however, because a person prefers a certian color is not a reason to act as if they are clueless.