Where are you gripping with your legs? A very common fault quite a few people have to pinch in with their knees. This creates a fulcrum and makes your lower leg swing back and forth. You should have an even distribution of pressure from your thigh, knee, and calf. Also make sure that you are gently sinking you weight into your heels, not shoving your heels down. That can bring tension through your ankle and make your lower leg shoot forward. Lastly, make sure your stirrups are the correct length! A too short or long stirrup will make it more difficult to have a strong leg.
And sometimes you can do all 3 of those things correctly, and when it comes down to it you just need to strengthen your legs.
Lots of work in the 2 point and without stirrups! Work up and down hills too if you can or across different terrain. Over time your legs will strengthen and be more still.