My horses go out with their halters on more often then they have them off. Never had a problem (knock on wood!) and they are breakaway halters.
My QH gelding used to go out without his, but he jumped out of the field into the neighbor's yard 2 weeks ago. He's had his on since incase someone needs to put him back if we're not home.