Katie, I totally understand. I have a TB mare as well, and she really responds well to gentle, loving people. I think she is very partial to women and nice men (haha). She gets kind of frenzied when people are harsh for no reason. When Jubilee colicked, the vet was putting a tube down her nose and she seriously moved her head an inch and he took out the stick thing with the chain and looped it around her nose.
One thing I really find with vets and farriers that although they are very knowledgeable and helpful, sometimes they don't know how to deal with horses well. Please don't misunderstand me, I don't mean that they don't know what they are doing, but in regards to handling a horse that could misbehave, they are sometimes rough. One of the first things the vet asked me was "Is she okay to handle?"
I think that part of it could be fear. Vets and farriers do a dangerous job. They are very close to horse's legs and feet, and they are probably trying to be very cautious as to not get hurt. Also, your farrier has never met your horse before and so he was probably taking extra precautions. I am not condoning him in any way, because I do think he was wrong the way he treated your horse and you are not overreacting. I would be concerned as well. But that's just a little bit of insight I've gathered from observing vets and farriers.
Anyway, I would definitely see about getting a new farrier. Like someone else posted, she could get worse and come to really not like him.
Good luck and I hope this helped some!