I always figure the younger the horse the shorter the memory. You will have to do some reminder work with him. I love Clinton Anderson's methods and use his "woking with foals, weanlings and yearlings" DVD's. Good stuff!
There are some rougher farriers out there too so don't rule that out completely. I had a farrier that worked on a foal from the day she was born till he had to quit for health issues. She never gave him a bit of trouble. I had been picking up her feet since she was tiny so she knew what was expected. The new farrier, she gave him hell, rearing, falling kicking you name it. I couldn't see any difference in what they were doing to her, she just didn't like him I guess.
"Until one has loved an animal, part of one's soul remains unawakened..."
- Anatole France