As MIE said, this is definitely not a dumb issue. To tell you the truth, keeping a horse perfectly straight is one of the hardest things we have to do in dressage. Once you've got straightness, you can work on so much more.
We can't have straightness before we have a 'forward button'. Unless the horse is travelling with impulsion and in front of your leg, it is very VERY hard to keep the horse on a straight line.
To start with, avoid working right on the rail, work on the second and third tracks as much as possible. I try to avoid the outside track as much as possible when riding no matter what I'm doing. The more you work on the outside rail, the more the horse will learn to rely on it to follow.
Ride forward and really focus on keeping your own body straight and in line with a point on the other side of the arena. Focus on this point and ride strongly towards it.
If we are talking primarily centrelines, this is an issue many people have - myself included on my current green horse. We tend to think of the halt rather than riding forward. My suggestion is to not put the halt at X in when you are practicing at home, just ride straight and forward the whole way down. Often a horse will drift one way more than the other. My boy for example, tends to swings his quarters to the left. To correct this, I apply a gentle pressure with my left leg, slightly close my left thigh onto the saddle and use a touch of left rein. This way, you are VERY slightly in a leg yielding feel, but not to the point where the yield can be seen. You are simply applying a little more pressure to the side the horse tends to swing out, to discourage the swing before it happens.