What Saskia said I would say is the best bet. You were "chasing" him away, and it's a horses natural tendency to be with their herd leader, which, he probably doesn't see the pony as a leader, so it's you.
He was trying to come back in, looking for signs from you, relaxed body, no eye contact and if you turned your shoulder to him, or your back. He most likely would have stopped on his own and maybe even came up to you on his own.
Lettin' the cat outta the bag is a whole lot easier than puttin' it back. -- Unkown