So I checked out the Equine Body Condition Score Chart. Other than the big belly he doesn't have the other fat deposits on the hips, back, thighs and withers.
When I go back and look at the pictures and video of him when we first got him, he has always had the pregnant mare look. He's had 3 wormings so it can't be worms.
Today we lead them together, starting in the pasture for about 15 minutes. It went well. We lead the little guy just behind the big fella and there were no issues. He was content to follow and the big guy was content to lead. About halfway through we switched up and lead the little one first and although slower the big guy was content to follow as well. Over the next week or so we'll up the time in the pasture and move it out to the road if they continue to do well. If so, I'll ride and get my lady to lead from the ground and then move on to me holding the lead from the saddle if it goes smoothly.