Flappy legs?

I reallllly don't know why, but when the pony I ride goes on a fast walking mission, my legs start swinging loads! I tried gripping more with my calves but the motion of him walking fast was just moving my legs around like a rag doll! My knees weren't pinching, I was making sure of that because my instructor pointed it out to me, but my knees were slapping the saddle/knee rolls!
I haven't been having regular lessons for a few months and have noticed things deteriorate with my riding. Mainly my lower leg :( But I am starting up weekly lessons again so should improve fast! (I hope)
Does anyone have any steady leg tips?
Or does this happen to other people?
He was doing the "we're on the last road to home so I'm going to walk as fast as I can now!" thing.

When there's a lot of swing in the horse's movment it's got to come out of the rider's body somewhere. Letting the lower leg be loose enough to gently flap is maybe not such a bad idea. If you tense it, then the motion goes to some other part of your body, such as your upper body, which is much worse to have "flapping about".

A video would be very helpful in seeing what is going , though.

I won't be riding him again till next monday, I will try take a video for you and I'll post it in this thread :)
It probably doesn't look as bad as it feels (I find this happens a lot when I watch videos of me riding!) But it would be good to see if anyone can point out where I need to improve

I would say heels down? Put your weight in you heels and see if that keeps you steady. My lower legs flap about when I ride bareback but if I push down in my heels it keeps them in a good position and doesn't kick my horse :o) Hope that makes sense!