First off, I say loose the spurs. If you need more forward motion, use a crop for now.
You need to stabilize your lower legs, and the best way I know of to do that is to ride without stirrups, LOTS. At the trot and canter when you're comfortable. Post without them. 2 point without them.
Also, your heels may not be down enough. When you apply leg, not spur, you should be squeezing with your calves, your heels still down. When you need the spurs. the action should be more of an inward rotation of your ankle, putting your toe out a smidge, maybe bring your heel slightly up if you need the oomph (you probably won't).
If your spurs have a really long neck (I think that's what it's called, the bar that the "jabbin' end" sits on...), it will be harder to keep it away from the horse's side when applying leg alone.
I hope that gave you some ideas, pics would help to see of the problem is your leg position or the design of the spurs.