The hair helps protect them from a flat-palmed impact like a good slap. But there's nothing wrong with being able to hear it! I would never abuse a horse or treat them cruelly, and I err on the side of caution, but if that slap doesn't hurt me it doesn't hurt the horse either.
I'm a total weakling, and so my instructors have to tell me to hit the horse harder with my crop when I'm riding. I usually have to turn my crop around so I can summon the necessary strength! The smaller impact surface makes it a stronger force (and it needs to be; I'm giving an aid, not praising them, when I use my crop) but I gauge my action so that you hear a slap on the horse, and possibly a slight whoosh if I'm standing still - it gets drowned out by the sound of the horse moving, I'm talking very quiet sounds - and never a whistle. Any lighter and the horse flicks its tail as if a fly landed on him!
Can you tell that I've spent a while trying to gauge how hard to hit by slapping the crop against my leg and the wall? XD
Slapping a horse on the head? Absolute no-no. Slapping them on the rump? Could be mistaken for aids. Slapping them on the neck or shoulder? Just another way of showing affection and praise.
Edit to add: Perhaps we're thinking of different kinds of slap here? A good hard slap is not the kind of thing that would knock a person down, and when I say it makes a good sound, it doesn't echo around the barn - you just hear a good clap.
If you believe everything you read, better not read.