Deworming a rescue is always a delicate situation. They can't really gain with a gut full of worms but deworming can cause problems too. ****ed if you do...****ed if you don't.
If he isn't in pain and just isn't eating, I would wait for the vet.
Rescues are challenging and this one sounds pretty bad. It is always hard to say but find comfort in two things. He can either get better or worse. Either way, he has known what it is like to be cared for -and- if it gets worse, you can give him the final gift of kindness and mercy.