Firstly, I don't think your horse remembers that specific incident, or that it's causing his current behavior. It's probably more of a bad habit, developed over time.
You have to be patient, and let him know you will not give up, no matter what he does. You can even give him a (gentle) nudge in the belly with your hoof pick as a correction.
Practice, practice,practice! Pick up his feet, and just holding them, without brushing. Then let them go, and start over. You can also hold a crop while you do this.