Even being wet is not the worst thing that ever happened. I mean, in the wild they get wet and the wind blows. But if the horse can move around, eat and find a wind break, it will usually be fine (as long as it is not clipped)
The horses usually have rather long "guard" hairs that kind of clump together and form a point that channels the water to drip off their sides, and the water does't go under the guard hairs to the under coat, which stays pretty dry. If you brush them while they are wet on the guard hairs, you have broken the seal, in a sense, and now water will go to the under layer and make the skin wet.
If I bring Mac up from the field to ride and he is wet, I brush only the area where the saddle pad will go , so as not to disturb the guard hairs that are channeling the water off his back and rump. He looks really filthy when I ride him out, but under the filth, he is dry.