It really depends on what you do. Recreational trail riders or pleasure riders aren't really athletes. Its more so the professional barrel racers, ropers, jumpers, etc that are athletes.
Competitive riding is a good workout, but taking care of a barn/horses is a workout in itslef, too. Lifting hay bales, grooming horses, bathing, lifting saddles (esp. the big heavy Western saddles!), sweeping, etc. are all good workouts. It takes a suprising amount of arm strength and elbow grease to curry and brush off a dirty horse. Same with hay bales (the rectangular ones)-they don't look heavy but they are densely packed.