I guess its like anything that is also a recreational pastime. Playing catch in the yard may not be called a sport, but a regulation baseball game is. Does that make sense?
Same, a leisurely trail ride is not the same as a competitive endurance or trail ride. The later may be called a sport but the walk through the woods would not in my opinion.
I guess if there is no competition I don't consider it a sport.
"Until one has loved an animal, part of one's soul remains unawakened..."
- Anatole France