I do leave mine on. I live right on a main highway, and if my guys were to get loose I would want a handle on them. Partly it's habit from my first horse who could be a little demon to catch when the mood struck him right. However, they do live in small grass paddocks without anything to get caught on, and are being looked in on all day and are stalled every night. If I were to turn them out into our 40+ acre cow pasture (with dense woods, creek, and pond), or leave them out all night unattended, the halters would come off, sure thing.
I've seen neighbors' horses running loose around the highway before, with no halters and refusing to stand still to put one on. Not a pretty sight. Well, the horses were pretty, running free as the wind, but you all know what I mean.