I leave breakaway halters on when I move them in other field for couple hours to graze. Mainly because I don't want to put them on in dark as I don't see very well. In main field (and attached grazing field) they never have halters on.
We keep halters off unless a horse is being led or in the cross ties. There are hook racks on every turnout gate to hold the halters/lead ropes which keep them convenient.
We have a bucket of grain right inside the barn to catch escapees. All the horses know what's in that bucket, just open the big arena gate, shake the bucket and slowly head in, they'll all come runnin!