If anything, I'd say the complete opposite. As long as your not straining your self while riding, then it is healthy exercise that could arguably keep you mobile for years longer than if you spent your young years sitting on your butt.
I'm not old, but a while ago I developed knee and lower back soreness... but I'm pretty sure that horse riding wasn't the primary cause. I went on a break from riding for a little while and have recently returned to regular riding this year and I no longer have either pains. So... yeah ;)
I don't think horse riding will cause any major damage in the long run. The riding gear available these days is very good in terms of posture maintenance and comfort, and if you ride in a good position you should be fine.
Besides, I can think of worse sports when it comes to causing pain in later life. If you've ever seen Free Running (where people run through streets bouncing off walls and flipping off high places onto solid concrete)... that's bad. Those guys are going to need multiple knee reconstructions before they are 30
He knows when you're happy,
He knows when you're sad,
And he always knows when you have carrots.