a number of years ago we had a transformer on the powerline by the highway blow. This is also right next to our pasture. The resulting fireball sufficiently scared all our horses to the point where any horse that was in that pasture will not for love or money go into the area where the fireball was. Our one gelding will just stand in one corner at the farthest end when in that pasture. In the space of two years he had a car come through the fence (for a horse that's afraid of tractors that was a big deal) and then this fire ball. I can't image how they would handle a genuine barn fire. They would never approach the barn again!
I agree that when possible halters should be left off. Then we had the car come through the fence and 8 horses escaped sans halters. The police showed up sirens and lights blazing. This resulted in a thrilling chase 5 miles down the highway, dodging both laypersons and the odd 18 wheeler till the horses split off to a quiet country road. Where they could be corralled, haltered and trailered back home. Now the horses wear halters. You have to weigh the consequences of halters vs no halter.
The best thing in the end is having a plan. You don't want to be sitting there at 2 am trying to figure out what you are going to do.