I'm coming at this from a different direction...
So, if your trainer/instructor heard and did not set straight the "mom" comments...inform your trainer/instructor of what you are going to do since you want the record straight on facts.
Trainer/instructor will either step up and step in and go stop the rumor mill or you will.
Now get your anger and emotions under control.
You in a weeks time go and speak as a adult to them one-on-one and educate them on what your horse is...
A look and stated, "I clearly heard you speaking of us, about us and I wanted to set the record straight with facts." "My horse is a solid riding partner. One who has never been aggressive nor does he have dangerous habits. He lost sight in one eye..
Many one-eyed horses exist, are perfectly safe and succeed in riding endeavors of Olympic competitors, high-end show horse, therapy horse....to backyard family horse, trail riding horse and many other jobs done safely.
If they were "dangerous" to ride, to handle or be in a ring with others they would not be in such a position of trusted mount."
Enlighten and educate...and also ask/tell them in future if they have a question or comment about you or your horse to take it to you not spread unfounded rumors please.
You will gladly speak with them to alleviate their concerns and expand their knowledge when it comes to your horse or your own self and their knowledge lacking.
Expect mouths to hit the floor, great embarrassment or should be and stammering... DO NOT
bring up their horses or their children's abilities or lack of... Mothers are fiercely protective and nasty at times when told of their "beings" downfall!!
Now be the adult and walk away...
Expect some flack and if they keep their distance, so be it! Just do make sure your trainer/instructor knows first and give them a chance to straighten the mess
...if they don't or try to push this aside, the answer is no and then you will deal with it...
owed more than to be maligned or your horse for their lack of knowledge...negative talk is cheap.
Do so though as a adult would...maturity present, emotion and anger put away till later reach out to these people to educate them. And no grudges held against....that is the tough part.