It sounds to me like your horse is young, has not been taught self carriage of his own body, and is overcompensating for a weaker side. Many horses have this happen, they are either right or left "handed", just like people. For a horse who is veering off in one direction means they are dropping the shoulder and do not know how to balance themselves. You may or may not be contributing to the problem with your body position, but as the rider it is your job to help the horse learn self carriage and how to keep his body in the proper position. Use your seat and legs to prevent him dropping the shoulder and veering off in that direction, and if necessary find a trainer to help you balance the horse and/or yourself.