I have been irritated by the opposite- the gaited horse people acting like my Mustang is too slow. And then if we trot to catch up and their horses go ballistic (because they don't get ridden enough and have a lot of energy) then everyone gives ME a dirty look because their horse can't handle my well-behaved horse trotting. Why should I apologize when my horse is well behaved and under control and theirs is not?
But, I do relate to what you are saying, as I have also owned Arabians and the QH people were always giving me flack about my Arabians. They were wonderful, safe horses that took me everywhere. And personally, I like a horse with a higher head carriage and a little "go" so who cares what other people like? Those kinds of people are usually afraid of their own horses so they have to pick on yours.
My cardinal rule is my horse only has to please ME. It is my horse, so it pleases me. I don't care if it pleases anyone else. I will change who I ride with before I change horses based on what someone else likes!
Currently I have a Mustang and a Missouri Fox Trotter, so I straddle both worlds, and it still irritates me that the gaited horse people (I ride with anyway) are always in a big rush and everyone has to go fast, fast, fast at a flat walk. Sorry, the Mustang doesn't have a flat walk, so we will periodically have to trot to keep up. Sorry if your horse can't handle that. Maybe you need to work on that, ha ha! (I do try to ride the Fox Trotter most of the time when I go with the gaited folks, but I also rotate my horses, so sometimes it's the Mustang's turn to ride).
My best friend whom I ride with all the time has Quarter Horses and we match up great no matter which horse I ride. She never gives me flack about my Fox Trotter. We simply pair up my Fox Trotter with her faster QH, and my Mustang with her slower QH. Works great!
A good horse is a good horse is a good horse. And both the go-y horses and the laid back horses have their strengths and weaknesses. Not to mention not all the horses got the memo and live up to their breed sterotype either! My very first horse was a slow, lazy Arabian.