You could try just hanging out with him and doing nothing. Go out and sit in his pasture and give him treats every once in a while. Also try to do things that he likes to do, like take him hand grazing, going for a role in the arena, etc. That's how I got my crabby gelding to like me. If you can, take over all or most of his care for a while. If you're the one who consistently provides for him and he knows your routine, maybe he'll start being nice again.
And if you ever see that hand again, slap him into next week.