First, I'd wait until shedding season is done. There is nothing worse than going to a ton of work to clean something, only to have it get dirty/hairy again 2 seconds later.
Then, I've had luck laying the pad on the ground, and using a really really stiff brush (one of those boar bristle kinds, I think...the kind you want nowhere near your skin, haha) to brush the hair off. It does take a while but it will do the trick.
Good luck! My horse is a gray too and those white hairs show up on EVERYTHING. Such a pain.
Fabio - 13 year old Arabian/Lipizzan gelding
Rest peacefully, Lacey.