I wouldn't worry about it at all. Your horse just sounds inexperienced. He'll learn, just give him the time to figure things out for himself. In the meantime assist him with the guidance he needs. All animals tend to prefer the easiest place or track to walk on and follow. Your horse will do that too when he becomes a little more experienced unless you tell him to put his feet elsewhere.
One thing that I was thinking of, are you sure that it's not you who are unconsciously cuing the horse to wander off the trail, by slightly turning your body when you are admiring the landscape or looking for something. For fun, try to let go of the reins or even better, ride at home on safe ground without a headstall and see how much or little/subtle cues it takes to ride the horse with the body. But concentrate on the surroundings not on the riding 'cause that's what you probably do when you are riding outside the property. You might get an “aha” experience..