My first and second horses were Arabians. I got a lot of flack at boarding barns for owning Arabians.
One day I realized two things.
1. The people that spoke the worst about Arabians being "crazy" were people that didn't even ride their own horses outside an arena!
2. The proof is in the pudding. I rode those Arabians all over city streets, about 1/2 the time by myself, and they took good care of me.
After that I decided that my horses only had to please ME. They didn't have to please anyone else. They are MY horses, and as long as I am happy with them and feel safe on them, that's all that counts.
I had another instance some years later on my Mustang. A lady I rode with was griping about how slow he was (he did walk slow). And then she followed up with a comment like "but don't sell him on my account." How rude is THAT? That was the best horse I ever owned. I didn't say anything but what I was thinking was "I will quit riding with YOU before I ever sell this horse."
Riding the Mustang was awesome because HE was awesome. I rode with a friend with $10,000 horses and my Mustang was never outclassed. He was every bit as good and any of hers and she knew it. She even borrowed him when she had guests.
So a good policy to adopt is "My horse only has to please ME."
PS. Another thing is, no matter what breed you own, there will be someone out there who doesn't like it. I've owned Arabians, the Mustang, Fox Trotters, etc. There is always someone out there who will dislike your horse simply because it's not their breed of choice. I really don't understand it because I see beauty in almost all breeds, but other people seem to stick blinders on and stick to one favorite breed.