I had an interesting riding lesson one day. The gelding I was riding had lived a very sheltered life (my family bred/raised/trained him so we intimately knew his whole life since he first hit the ground). There were many things he had not seen before, and a black and white pinto was on the list (actually, he had never had any exposure to a pinto of any color nor had he ever seen a cow). So, there happens to be a black and white pinto that had a stall and run by the arena. We were riding in this arena with no problems, until the monster left the barn stall and was visible. He was totally freaked out, would not get anywhere near that horse of a different color and would not take his attention off the scary monster. He had no problems seeing the other solid horses coming in and out of their stalls/runs. He had no bad experience to associate with a pinto of any color (or even a cow). He was not responding to me or my prejudices against loudly colored pintos. He later got over his fears when he began to compete in 4-H and when my mom bought a black and white pinto mare years after he saw that they were just other horses.
I see no reason why a horse might not react to seeing a person of a different color for the first time. Only based on color tone and nothing else. I am sure that most horses don't freak out when they see a different color horse for the first time ever (but mine did LOL).
This same gelding I was riding that was freaked out for most of my riding lesson, well his mother also had a horse prejudice. If she saw a black horse, any black horse, she believed without a doubt that they were a well endowed stallion and she was going to show off to attract their attention. She had never been pastured with a black, but had been bred twice to a black QH stallion. This behavior towards blacks bugged my sister to no end (showing in 4-H was a pain) and so she went stud hunting. Found an inexpensive breeder to breed her mare to, they owned two studs, one grey and one black. The black was a better looking stallion, but she couldn't continue her mare's prejudice, so she bred to the grey. The mare's black prejudice was gone, and my sister no longer had a nutcase mare when she saw a black horse.
Side note: I have had complete strangers from Russia or the Ukraine to to talk to me in the native language. Even when I tell them that I have no idea what they are saying, they usually keep trying to talk to me in a language other than english (some boys my age even laughed and continue to talk talk in a foreign language)... Apparently my Scandinavian (Norwegian and Icelandic), German, and Irish heritage results in looking like I speak English as a second language instead of my only language? Even teachers I had in high school thought I was yet another immigrant from Russia or the Ukraine. Probably didn't help that I have a Minnesota accent and had just moved to Spokane, Washington where there was a large immigrant population. But still, my first name is German (I believe) and my last name was Irish, yet based on looks and accented voice even the teachers thought I spoke Russian. LOL! I took it in humor, but still, it really throws you off when someone wants you to translate for them in a store. Um, yeah, sorry, only understand one language and only a few words/phrases of other languages