I can relate to this totally. We don't have any extreme trails here in Latvia (as our terrain is mostly flat, though some trails go along steep river banks and such), so most people act as if trail riding is just hacking lazily on just any horse, and that even people without riding abilities can do it. Well, I don't agree. My horse has to tolerate loud, sudden noises, wildlife (moose, for example), traffic, change of environment (if I take him to the sea), loose dogs and everything else you could experience out in the forests, and do it all with patience; he has to have 100% of his attention and me, change gaits instantly (pits, barbed wire, hidden in moss in abandoned warsites, etc.), has to be endurant, work well with other horses, stay with me if I happen to hit the ground, and so on. I am lucky that his initial training included mostly trails, not arena work - he is a perfect trail horse for the trails we have here, besides, both of us get bored in the arena quite fast. As for myself - trail riding needs lots of fitness and the ability to react quickly to any changes or obstacles in the trails. It calls for lots of awareness and attention, which, to my mind, cannot be trained in an arena where everything is almost the same every day.
I have come a long way, to surrender my shadow to the shadow of my horse.