Haha welcome to my world...... except my horse is white 78% of his body :P Honestly I find that my saving grace is baths every other week during the summer. I just use mane and tail and I SCRUB like no other, which he loves. I usually rinse first, then soap, then rinse against and towel dry. Then during those 2 weeks of no bath, I make sure to use something like Laser Sheen (<3) or Cowboy magic to spot spray and leave it sitting while I work on something else like his mane. When I come back to it, it's dried on enough that I can curry and it falls off easily.
My weapons of choice: metal curry (for cleaning the brushes OFTEN), rubber curry (to scrub the face, legs, and the other side for the body), medium stiff brush to flick all the dirt off, and finishing soft brush to add shine and suck up the remaining dust.
It definitely is a work out, but I do it everytime I ride him and he is always GLOWING white. He can roll 5 times in a row but I always get all the stains out.
Don't curry wet mud though, if it's wet then you can hose it off (granted it isn't winter) and if it IS during winter, I usually spray laser sheen on it, leave it to dry, curry... towel... spray, curry towel.. and it gets it right off though.
I've found that the less gunk on their actual skin (the base of the hair follicle) the easier to get that dirt off.. which is why I LOVE every other week baths :)