Most green horses do not walk in a straight line for quite awhile. Sounds like he is very green. I always see it as a positive thing because green horses that go straight are usually very stiff and need tons of bending to loosen up. It sounds like he needs to be taught basic cues. The way to teach a horse to go straight is through bending. It's counter-intuitive. Teach him to follow your cues and do simple turns, then changes of direction. When he has done those for awhile and understands your cues you can work on circles and serpentines. When he can do a perfect, fairly small circle in both directions he will start moving in a straight line. Be sure to work on slowing down and halting before you take him out on an exciting ride because I can almost guarantee he doesn't understand that yet either.
Sometimes riding a green horse feels like riding a snake or wiggly worm as they bend their body back and forth and weave their way along. If you take them on a narrow trail when they are first starting out be prepared to brush the bushes on either side of the trail. And remember, if they don't know how to go straight at the walk, they certainly won't go straight at the canter either.