I'm also a plus sized rider (just got back in the saddle after 25 years out of it!) and on my first lesson last week I was nervous about the whole riding again but also my weight.
I'd rang ahead and given my weight so they were prepared, but after the lesson, I was congratulated on perfect posture/seating and the instructor explained that a heavier rider who has a good seat will do less harm to a horses back than an 8st rider who doesn't know what they're doing and bounces around.
This makes me feel better about being a plus sized rider as the last thing I want to do is harm any horse (even though i'm working on losing weight - it's not going to come off overnight) and she's told me to carry on and be confident that i'm not causing the horse any harm at all.
Horse riding doesn't just have to be for skinny minnies
but I do know what you mean about the looks - I've also got those when looking for riding wear................But am I bothered - not a bit