How old is the horse? And, for how long do you ride?
I ask because young-ish horses can sometimes be like little kids. They wander about, their attention being caught by... only they know. I used to love the comic "The Family Circle" when they had a cartoon about the mom or dad asking one of the kids to get something or deliver a message and the drawing would show the 8 times longer route that the kid would take. Young/green horses can be like that.
And, if you are not riding the horse until it is tired, you a) aren't developing a real good work ethic in it, and b) it isn't learning to conserve energy for what the day might bring. Some horse owners may disagree (we do that!), but I really see a benefit in both of those traits. I'm not recommending riding a young horse until it stops from exhaustion, just long enough until it isn't thinking of anything else but one foot in front of the other. They relax, don't make up directions to take on their own, etc.