Some horses are just more sensitive to fly bites than others.
My TWH gelding absolutely hates
those big, nasty horseflies that come out around late summer. And, unfortunately, they seem to LOVE him and are drawn to him like a magnet even when we are riding with a large group.
I've been on some rides where I felt like I should just turn around and ride backwards in the saddle for as much as I kept having to turn and smack those flies off his rump. He's not bad. . .he doesn't throw a total tantrum or buck, but it's a miserable time for him.
One thing you might consider is Fly Armor - either the bands that fit onto your tack, or the kind that can be tied into the mane and tail. I've found that they are the most effective thing, especially in hot weather, because unlike fly sprays and spot-on fly control, the Fly Armor actually works BETTER the hotter it gets, and it doesn't "sweat off" or wash away in a stream crossing. I started using the mane tie-in band late last summer and it really worked well for us.
Aside from that, I don't like my horse to get too dependent on me to take care of his fly issues. Yes, I will swat them away, but he doesn't get to cut his work short just because he's reacting to the flies.
I expect my horse to stay focused on what he is doing and where he is putting his feet when we're out riding. You may want to consider a fly bonnet, or rump rug, or some other more physical/less chemical methods of fly control for your trail rides in order to keep your horse's mind off the bugs, and on you.