I'm a similar height to you, SL, (and VERY slender, I'm only 110 pounds maximum) and I'm pretty much the same! I won't ride a slender horse smaller than 15hh because I feel too big, but a chunky horse or pony, I can go as small as 13.2 and feel and look perfect. My Monty is 15.1hh and has a 2 inch wither on him so without the wither he would be 14.3, and I am just the right size for him.
I used to ride a 12.1hh pony and he was the absolute best, but just too small. I was in a saddle too big for him and it was at least a size too small for me by the time we sold him. I can't ride effectively in any saddle smaller than 17 inches (english size) because I have REALLLYYYYYY long legs, and saddle size equates to the length from hip to knee. On me, that's 17 inches. Any smaller a saddle and I end up having to ride with my stirrups too long, or in the wrong spot in the saddle (onto the cantle more than onto the seat). I can go a half-size bigger but not a full size. And I am TINY so most people try to tell me I shouldn't be riding in such a big saddle - until they see me ride in it. I had a saddle fitter tell me I shouldn't ride in any smaller a size than my saddle (17.5 inch, but it rides small for its size, if you know what I mean). The funniest thing is, it's mostly dressage people who try to tell me I'm in the wrong size saddle - jumping and eventing people seem to know about saddle sizing. That's a huge generalisation but it's what I've noticed. One of the top dressage coaches in the state told me to just ride with longer stirrups, but I can't - I already ride REALLY long because my saddle encourages (demands) a long 'classical' leg position, and if I went any longer, I wouldn't be able to reach the stirrup.