A number of possibilities come to mind. Does it happen with others too? How about when he's not being ridden?
- The horse could have a physical issue that is causing him to walk like that. The horse should be looked over for pain and perhaps evaluated by a vet for a neurological issue.
- The horse could be unbalanced and trying to adjust to you.
- As a beginner rider, you may not be strong enough in your aids (reins, legs, seat) to keep the horse straight, so his attention could be wandering around and then his body is following. You have to ride straight to help the horse be straight.
* I'm often reading and posting from mobile and Siri loves to make a mockery of the English language.