Is it more clay than dirt? Clay can really get stuck on tough. Normally, I would suggest a metal curry or rubber curry, but you've already tried that. Since you can't give a bath, you could try "toweling"
You will need a bucket of water (hot is better) and a towel. Get the towel wet & wring it out a bit, then continuously wipe at the dirty spots. The goal is to not get your horse very wet, but clean the dirt off. It is very similar to how you might wash your face with a washcloth, but don't use any soap on your horse.
When you are finished, you can use another towel to dry him up if needed.