If he's lunging just fine he doesn't need to be desensitized. I think people get very focused on that and imo overly so.
Think about the word "de sensitized".
I feel all you did by desensitizing him is teach him not to move, you want them to have a "healthy respect" for the lunge whip, no you don't want to be fear. You also need to mean it with your body language. Think of the whip as an extension of your arm, use to convey your meaning and to "reach him" if you need to.
I'm sure your "conversation" right now is along the lines of "please" "no" "please" "no", etc. What is should be is "please" "no", "next time I won't ask so nicely" "hmm" "ok, fine, GO" and make him go. You aren't getting beyond asking very nicely with very quiet cues right now and he's learned it's ok to ignore that.
He does sound like a nice horse, but don't be afraid to push him a little, he won't break.