I was also taught a whip is for "Forward!" while spurs are more for refinement.
I know horses can dull out to spurs faster than to a whip, but it all depends on how these tools are used. It's jsut that so many riders apply leg with nearly every stride. Nothing could dull a horse out faster then if you did this with spurs on.
I think several folks keyed into the point that if your horse freaks out with a whip/crop in hand, then this is something that needs to be looked at as a place for desensitizing. He might be reactive to a lot of things, so maybe you should look at that and see if can tolerate things above and around him and such. I agree that work on the ground with a dressage whip in hand may help here.