Reiterating what everyone has said, every horse (just like every person) is different. Dallas loves to have his face rubbed like you might take the hair off. Jasper doesn't like anyone going near his face, but he loves a good stroke-n-scratch on the neck. Nugget just about falls asleep getting groomed. Toby likes to have his face scratched after getting his bridle off, and loves a good butt scratch (like, right at the base of the tail). I do pat his neck when he does well, but also rub his neck or, usually toward the end of the session when stretching, lean over and scratch between his ears from the saddle.
I do think a soft pat is different than giving them a swat to the shoulder or butt, and they know it's different, or at least most of them do. I think a lot of the been-there-done-that-lesson/trail horses in particular are sort of aware that we're kind of strange and they meet us halfway.