I lease a horse who has learned that trick. He used to do it when I was taking him out of the field, away from his dear mare friends. he no longer does that, but he still does it when loading him into a trailer.
The thing is, I have to have his thought WITH me every second of the time I am leading him, or loading him into the trailer. He is quick as greased lightening. He can go from following along, to a break away, ripping the rope right through my hands, in a second flat.
Using a rope halter helps, and having the ground work done, and redone, and reminded and reminded, every time I lead him, keeps him on his toes and thinking about following me. I also watch his eyes. The instant I see him looking away, I am giving him a bit of a jiggle of the rope to get his attention back where it belongs.
your hrose is a baby. he cannot be expected to NOT have some issues. you should expect them and work to teach him otherwise. My horse knows he should not barge away, and only does this if he can get outside of my attention.
A baby will not necessarily KNOW what he is supposed to do .
And, I agree that him not having outdoor grazing time must make him just crazy to get out there and run free. That sets up a difficult dynamic as it is.