Like BackInTheSaddle said, some people are just insecure jerks that need to find something, anything, about another person to make fun of to make themselves feel better. I am only a tad overweight, but sometimes I feel absolutely huge. See, I LIVE with one of those people. My younger sister by a year is very pretty and very in shape, and is the biggest b*tch you're ever going to meet. She's vain, selfish, mean, and unsympathetic. She calls me all sorts of names, usually unprovoked, just to get a reaction out of me, just to make me feel like sh*t. Because underneath that pretty face is a little girl who feels insecure and feels like the only way to help that is to make others feel worse. Not that she deserves any sympathy for that. But just so you know, no matter what you looks like people like that will find ANYTHING to use against you.
Don't ever think of giving up something you love because of what others say. They just aren't worth your time. It's easier said than done, and I battle with self confidence issues and anxiety attacks daily. But you just have to keep you chin up and keep reminding yourself that you are an amazing person, and a great rider with a passion for riding and horses that none of those people will ever have or comprehend.
I give myself very good advice, But I very seldom follow it