I have to agree with bsms about the aerobic part of it. Riding is not an aerobic workout, it is more like strength and balance training. I wouild likely compare it to something like yoga.
I can attest to that completely. I can ride from dawn until dusk at all 3 gaits, riding different horses that travel differently, getting on and off and sometimes fighting with them a bit. Even after all that, I'm not tired at all.
However, I've spent the last few days putting up new fencing around my property, digging post holes with manual post hole diggers, tamping down the dirt around them with a ~30 pound sucker rod, driving t-posts with a 40 pound manual driver, etc. I am so sore and so completely worn out right now because it's more aerobic exercise than I am used to. Of course, it is strength training too, but I had to stop more often because I was out of breath and my heart was pounding than I did because my muscles were tired.
According to the calculator thing, I've been burning around 2500 calories every day I've been doing that. It's no wonder my pants are feeling too big
...especially considering I usually only eat once a day and that totals up to maybe 1000 calories.