If this horse is 100 lbs underweight, I do not think that working him hard is a good idea.
Magic Melly- if your horse has been alone for the majority of his life, he may also be suffering from sensory overload and the inability to communicate with his kind. He may be excited to see others of his kind, but he may not know how to interact correctly with them and so he is fretting. This is sort of what our TB filly is going through. She's had a tough life and needed to be quarantined for a few months due to a virus, then a bad injury, and I don't think she had any companions at her old home. She grew up virtually alone, with no communication, no friends (she was also an orphan, and missed the vital teachings that a mother gives it's foal). So when I started trying to introduce her to other horses, she'd be over the moon with excitement, whinnying to them, dancing, pacing, etc., but as soon as she got to where they were, she had no ide what to do. She'd just stand there, eyeball them for a few minutes, and walk away dejectedly when they chased her away (due to her lack of ability to 'talk' to them) or ignored her, and start focusing on a new horse in another place that she could see, and start calling to them, pacing, fretting, etc.
Your horse's behavior reminds me of her's though maybe a little more extreme due to whatever personality differences, environment differences, etc that they may have. With our filly, we've just tried to introduce her slowly to being with other horses of her kind. We're starting with a docile 'everyone is my best friend' type mare, and slowly introducing her to more dominant, demanding herd members. It overwhelms her sometimes and she'll 'freeze up' and sort of zone out to ignore them, but she is slowly learning to exist with other horses.
This, ofcourse, may not have anything to do with your gelding, but I figured it was worth sharing just in case he might have similar problems. Think of it as him being a socially awkward kid that needs some help adjusting and learning to communicate. If that IS his problem, moving him constantly, or switching his herd mates up won't really help solve the problem. He needs to feel comfortable in his 'home' (HIS pasture or a neutral grounds, don't bring him into another horse's field and expect him to fit right in) and have others introduced to him slowly.
Of course that's all assuming he was alone when he lived with you at his old home. If he grew up in a herd, this is very unlikely to be his problem.