To make sure I understand this correctly.
He only has loose stools when trail riding or doing some other activity that makes him nervous? Normally, while happily grazing in the pasture all his stools are firm, normal pile of balls, horse crap?
If that is the case (and I hope that it is) then I'd say it's nerves. Most horses will have more bowel movements when they get nervous. I've had them crap 3 times in less than a mile because of encountering things that they were very uneasy about. Takes time for them to get more comfortable. Some longer than others, but eventually they get use to the activity as it becomes more normal and the loose stoolst should become less common and eventual stop (if that is indeed the case).
If the stools are usually loose most of the time there can be a number of potential causes. Since worms have been ruled out I'd check on what he might have access to eat besides grass (e.g. Ingesting acorns, oak leave, etc... can cause loose, runny stools). Of course those are not the only potential causes, but you need a vet to consider all the possibilities. They would have the means of checking things beyond the external and the obvious items.
Good luck. I know how frustrating it can be having a problem with no identified cause or treatment.