The reason I say I'm "scared", is because I am more anxious/paranoid than actually scared about it. I've been riding for 5 years, and the fasted I've gone is a hand gallop (A borderline gallop, if that clarifies). Before, I never had any anxieties about it. I LOVED going fast, and I couldn't wait until I could ride again and lope around.
I haven't had a tramatic fall or anything. I've never lost control at a fast pace. Now if I even consider trotting, I phsyc myself out to the point I just don't do it. Once I get going, I'm fine; but it takes me a lot of pep talking to convince myself to just go faster.