It's a little hard to say as you haven't said how long your horse has been under saddle. The behaviors you are talking about seem like "green horse things" If so, consistent, slow, quiet work will fix it. When starting a young horse I always carry a crop in my outside hand when I am first working on circles. Then, as I use an opening inside rein and keep the outside rein against his neck I will tap on the outside shoulder when my horse starts to deviate off the circle..usually they bulge out through their shoulder. I also always teach things at the walk before I try them at the trot.
You have said you use your outside leg. That is good. You can also try a bit of a counter bend (bend him out and push him in) on the circle when your horse begins to bulge. That also helps to catch the outside shoulder. As soon as the correction works allow your horse to go back to "straight ahead" on the circle.
As far as wandering or weaving a bit on the straight lines, make sure you are riding forward enough at the gait you are in. It is easier to stay straighter if you are going forward. The more you "slow down" the more weavy you usually tend to get. Also, the work you do with correcting your horse on the circle will actually help him stay straighter on your straight lines