One of mine was like this when I first had him we discovered that he had a problem with his neck and was in fact in continuous low grade pain. Regular visits from the chiropractor eventually fixed this and he became very easy to maintain.
Though you say his worm count was nothing amazing I would be inclined to do what my equine vet suggested with any horse that is not thriving well and that is to 'bomb' drench the horse.
You worm the horse once a week for three weeks with a wormer that contains Ivermectin and Praziquontal. Sometimes a horse will get such a large burden of worms that some will encapsulate themselves in the surrounding tissues and wait for a time when there is room for them in the gut. You worm the horse and the current worms are killed and pooped out. The encapsulated worms then immediately hatch out and repopulate the gut leaving your horse in the same situation.
1st worm dose gets rid of anything in the gut
2nd worm dose gets the newly hatched worms that had been encapsulated
3rd worm dose gets anything left behind from the previous two wormings.
This had proved a great success on several horses that I've bought. They've immediately started to put on weight.
As for teeth - did the vet check them or did a specialist Equine Dentist check them. I have no faith in vets doing teeth. A vet is the equivalent to a general practitioner doctor and you wouldn't go to him if you had tooth ache - you'd go to a dentist.