I think you said it. He's not paying attention.
Your horse's attention is really, ultimately, what you are riding. In the arena, there is far less competeing for his attention, so he maybe gives you more, though I might venture to guess he might be going a bit on "autopilot" in the arena, which means he isn't really paying as much attention as hyou might think.
When you get outside, there are many more things for him to think about or worry about, and he is a horse, so he will be reacting to that by wanting to MOVE. So, one thing to do is let him move. Rather than holding and riding the brakes, so to speak, let him move. I don't think yo9u will want to go out there and start cantering around right off the bat, not if he feels like he is not paying attention. The faster his speed, the less likely you will have him paying attention to your rein or leg.
So, walk but go in circles. Keep him busy, busy , busy. YOu can stop, yes, and do, and get a good stop, whatever the rein contact required. But then move off and circle and move and change and keep him busy, always changing what you are doing. Go out for a bit, then back into the arena, then back out, then back in. Mix it up. Canter in the arean a bit to loosen him up , then go back outside and see if he wont go smoother and lighter. When you get him softened a bit and he'll stop pretty good, then get off and say, "nuff for today". He will get to think the rest of the day on what happened.