I vote bay. Something is going on, but my horse does the same thing and he is clearly a bay. But at times he makes me think brown, and I'd be more inclined to call mine a brown than yours. Test! I'm curious!
Late fall: Slightly lighter areas on the withers and between his hind legs.
Early Winter, bay, no light areas.
Just the other day: Faded "feathers",lips, nose, around eyes, withers. He's shedding.