I'd try transition work, both in the saddle and on the ground. It will also give you a chance to get your horse really soft (in case he's a bit jaded from his lesson work). Start with nothing but an intent to slow down. Then tell him super gently, by shifting your weight and giving a whisper of a tuck (not a pull, just a "signal") on the reins. Increase the signal until you feel the chance you wanted, then absolutely let him be and praise him. Sooner or later you'll trust that he'll respond, and you won't be nervous anymore. Ideally, you'd do this with an instructor or experienced horse person watching you for a bit before flying solo.
You can do the same thing on the ground: changing speeds at the walk. First, simply fall back a bit, but still on a slack lead rope. He'll match speed or not. If not, give him a little tuck on the halter, but STOP the second he slows even a little, so he knows this is what you were asking. If necessary, repeat after 2-3 seconds, so it becomes a new "aid".