I have the same issue with my horse, he is petrified of the main tack shed - he will stand just fine at the far end of the hitching posts, but take him any closer and he freaks. We are thinking someone jumped out of it or something (that's the best guess anyone has at least; whatever it was, he associates something negative with the shed). I would assume your horse had some similar kind of incident, be it wildlife or human-related, that made him afraid of the barn.
As far as the kicking and being antsy, leave him be for a while. I'd leave him for an hour tomorrow, an hour and 15 the next day, etc... basically "sacking him out" gradually. Don't take him out while he's kicking, wait until he's been quiet a few minutes. Definitely stay within earshot to make sure he doesn't have an epic meltdown though.
If the kicking persists after a few days, I'd probably give him a sharp "HEY!" so he stops, then walk right back out of sight (don't want him to get the mentality of "If I kick, she comes back to get me").