I had a very finely built gelding (QH/TB x) he was 15.2 hh but I always felt big for him. He seemed to hate being ridden out and I always felt like my weight was contributing factor. After I sold Comet I purchased the mare which I now have, Phoenix. PHoenix is a draft breed and weighs close to 600kg. She carries me effortlessly and I feel really good on her.
The other reason is a gf of mine has just purchased a new horse. My friend is newly returning to horse riding after quite a few years. I think this is a lot of the reason she has bought the horse that she has. It is a pony 14.3 hh so not overly short but it is SO petite. This mare is really finely built, the first thing that popped into my mind when I first clapped eyes on it was "it's tiny!". My gf is not a small woman either and I feel that she wanted something that didn't overwhelm her with it's size. I feel genuinely concerned about this little mare. We live amongst some pretty big country and my friend expects this pony to haul her up and down them.
I don't know. Maybe I am just putting my own feelings of self conciousness where they don't belong - on other people. I do wonder though does anyone else worry about how much weight a horse should carry?