always keep in mind that your horse has the attention span of a two year old toddler, at best. He will always be looking for something to focus on, and it's either going to be you, or something more important. He doesn't have an agenda to try and trick you out, or be a spoiled child. He just goes where he thinks things are more important. other horses and food are about the most important thing to a hrose, except a scary rustling in the bushes, and then only for a second til he knows it's the wind.
the trick is to have his attention, which mean you have to be both interesting and important. it's likely that after an hour, and if you are a moderate beginner, you did not realize that his attention was not really with you. it was probably not really all that "there" when the horses entered, and suddenly there is something very important there. He left you altogether at that point. then, it took a lot to get him back to you, mentally. As a greener rider, you are not able to be that important that he gives up the outward draw and comes back to you. Your trainer can, but even she had to work at it.
anyway, my rambling is only to say that when you are riding him, or doing groundwork with him, it can be easy to keep his attention if there is no competition. once the competition arrives, you have to be smarter to win . But, don't think of it in terms of "my horse wants to be spoiled, and has an agenda". He just follows his horsey brain, so give him something to follow.