I think it's plausible but the scent explanation is more likely.
I have one that doesn't like palomino (or other very light colored horses). Hondo was kicked by a pally mare when he was first used at stud when he was young. Still to this day, 25 years later he is terrified of light colored horses. I can recall riding him in a HUS class and passing a pally, I could feel him tremble and it was all I could do to keep him from trucking it across the arena to get away. He wouldn't cover any light colored mares after that but would gladly cover any other, it was a real pain to have to AI every palomino, grey, buckskin. If I bring Lacey (grey) in the barn, if he is in his stall he goes to the back corner and "hides" or goes back outside to get away.
I think it's possible. I've met horses with aversions to many different things because of experiences in their past, they generally don't forget. I'd want to know that horse's history though before making that call.
Life is like a camera. Focus on what's important, Capture the good times, Develop from the negatives and if things don't work out, Take another shot.