The only time I have ever worn gloves while grooming a horse is when it's super cold outside as my hands will otherwise go numb. Like someone mentioned, if you're that concerned about hygiene then the barn probably isn't the place for you. I don't see avoiding dirt altogether as a very good hygiene practice to follow- look into the hygiene hypothesis
Even if you're that concerned about it, washing your hands before eating should suffice.
As far as the implications to the horse though? I doubt it matters one way or another to the horse. I agree that a closer contact with the horse (without gloves) is beneficial to detecting something out of the ordinary, but if you feel the need to wear gloves then go for it.