When I'm working my horse in the arena, I will sometimes ride with my headphones in. But when I'm riding the ditches or road, I never do.. I have speakers I keep in my saddle bags that hook up to my phone, which I keep in my pocket. I don't use them on busy areas, but there's some dead gravel roads near me, where they're nice.
Its flat and boring where I live, with no access to "natural" areas without trailering.. So its all ditches and roads. When I haul away to trail ride, having music on never even crosses my mind because its so interesting.
If you're going to use them, I would only use one and make sure you can hear outside noises. Otherwise, I'd just get my horses used to the speakers.. No reason not to.