I think it depends on the saddle, the jeans, the rider, and the activity. Say, for example, a very soft French leather Butet, some tough & slightly baggy jeans, a somewhat off-balance rider, and a jumping course. I do believe the bunched up, baggy jeans could rub along the seams of the twist and cause damage. I've seen it many, many times. If you look on eBay, many of the nice french saddles (highly used ones) will have splits along the seams of the twist (or just major wear); this can be attributed to riding in jeans, among other things.
Of course, metal grommets on jeans pockets are also a no-no!
I always ride in breeches. I feel like I can't move my legs freely in jeans. (Or maybe my jeans are too tight--haha!)
PS-I think riding in jeans in a Western saddle is fine. They are built differently. And as others have said, people have been doing it for ages!
Monte - 15 year old Paint/TB
Auric - 13 year old Arab