The horse is covered in fur. I doubt he can physically feel the difference between skin or leather or cloth.
Even if he could, he's still a horse, and doesn't understand the concepts of "skin" "leather" "cloth".
If you're concerned about hygiene, then a barn probably isn't a good place to be. Even with gloves you really should wash your hands before sticking them in your mouth, or on food, or in your eyes, etc.
Wear gloves if that makes you more comfortable, but unless they're some kind of latex medical kind of glove, your hands aren't really that much cleaner for having been covered.
* I'm often reading and posting from mobile and Siri loves to make a mockery of the English language.