Gloves while grooming? Nope. I wear them when it's REALLY cold, but I prefer the ones that are fingerless with the little mitten top you can button over it - I need my fingers for the finer things like tacking up.
Otherwise, I *Like* touching my horse when we groom. I like to be able to scratch her and feel her fur. The horse doesn't care, but I do. I accept the dirt as a part of owning a pastured livestock animal - I think if you're worried that much about hygiene, like another poster said, a barn is maybe not the place for you to be!