I agree, riding is definitely exercise, but should not be the only thing that you do. Me being curious, but, did you ask the expert why it isn't? I'll give the person the benefit of the doubt, as boxer said, the fitness person was probably thinking of more of a cardiovascular exercise.
Lol, I half argued this in school. My college requires freshmen to take 2 semesters of Fitness and Wellness (A.K.A. Gym, lol). I did my darndest to insert horses into it whenever we had a project to do. Example: We were supposed to design out own circuit training routine. I came up with exercises that worked with horseback riding.
Anyone who thinks that the horse does all the work must have never ridden, and deserves to be pitched up on one and taken for a 2 hour trail ride over varied terrain. Even if they didn't change their mind, they would be too physically sore to tell you how little work they did.
A stubborn horse walks behind you, an impatient one in front of you, but a noble companion walks beside you ~ Unknown