Not at all... not sure why it would? I ride both, and would be riding western on Zeus as well as dressage and jumping if my western saddle fit him. I don't get why people dislike western. If you're riding properly you'll ride pretty much the same no matter what kind of saddle.
I only ride english, but it doesn't bother me at all if someone is riding western around me. However, it would bother me if western people are annoyed when I am riding english, that would just show intolerance on their part.
I ride both , somedays I feel like riding in my English saddle , somedays I feel like riding Western. It depends on what I am working on that day or what my mood is. I ride with others who ride either english or western.
Now that being said are my horses "trained" Western? Not exactly, they neck rein ( sort of) but we are still working on that. Do they have a slow western jog , well not exactly ...I can get my Dun to jog very nicely but my Paint is a work in progress (although getting better and slowing down more)
But to answer the original question it doesnt bother me at all if someone else rides Western or English .There is a lot that can be learned by both. I at one point had an instructor who taught me a little about dressage while I rode in my western saddle.