Yeah, there is a big difference. There is no excuse for hitting a horse with unecessary force or repeatedly, but if the bratty pony on our yard tries to barge me, he is getting a smack on the chest, and trying to bite gets him a swift tap on the nose. It not only stops people getting hurt but is beneficial for the horse too - horses that don't learn manners don't end up in nice homes, afterall.
It's all very well to be sentimental about horses, but they are never going to hold it against you for reprimanding them within reason - that's how they learn from each other in the field, and it's an easy way for them to understand when they're wrong. Sometimes waving arms and making yourself 'big' to warn a horse off something just isn't enough.