It is a very interesting question, and personally I would say something.
The only way that I can trust the "big girls group' is if they do actually tell the truth, because otherwise who can I trust. Example, there is a a girl who has a beautiful big Canadian Horse, he is always beautifully turned out, and carries his plus size rider with ease, so what's the issue? Well she was a bad rider, she did have coaches, but honestly it was hard to watch her ride, because, for example, she posted the trot by lifting on her hands and pulling herself up.
So many people said "Awww you look good together" and left it at that, not fair to have a go because of her size. Well that is not fair on the horse, if you are big, and you ride you have to have the right horse, the correct fitting tack for horse and rider and you have to ride well, or at least be working to improve.
So, I would not, could not ignore it, everyone can improve their riding, no need to be nasty, just matter of fact.