It's very individual.
I'm about 5'1 or 5'2, so quite short, but personally I prefer larger horses. My gelding is 15.1 and a half, and my filly is 16hh+ [not 100% sure, haven't sticked her]. I can ride anything from about 12hh upwards, weight wise, but anything under about 13.2 [depending on the back length of the individual pony] I have to ride bareback because I ride in quite a large saddle. I cannot ride in anything smaller than 16 inch and that is REALLY pushing it, my personal saddles are 17.5 inch [synthetic dressage and jump] and 17 inch [leather Stubben dressage] and the 17 inch is a teeny tiny bit little for me but not so bad that it's a problem.
I might be tiny and very slender but I have very long legs/thighs for my size.
I am most comfortable on a galloway [14-15h] or a horse 16h+; for some reason anything between those heights feels "the wrong size" and I feel unsafe. I don't trust my gelding even though he is a very safe ride 99% of the time and knows his job, whereas I do trust my filly who is not quite broke yet [ten rides I think??] and highly sensitive, and can be volatile. And has bucked me off. AND is, I think, cold-backed.
The photo I have attached, in show attire, is of me on my gelding, with stirrups two holes shorter than usual so that I looked like I fit him better. He is VERY chunky, you would never guess he's an Anglo Arab. I get asked regularly if he's a QH.
The photo of me in the vest on the chestnut is of me on my filly, stirrups at "normal" length. Excuse the chinstrap of my helmet, I did it up as soon as I realized. She is 16hh+ and quite slender. I attached two pics of her to show the difference between when I first get on and when she's had some time to warm up/get used to the idea of me being on her back, she humps up SOOO much!
And then the photos of me on the little grey, I sold him in... early 2011? And grew an inch or two after that. He was 12.1, and I was wayyyy tall on him, but never got asked about it because I'm so slender. Nobody ever believes me when I say I'm about 109lb. First photo at dressage length, second photo over fences [excuse my shocking position, it was nearly 3 years ago now!] and third photo at jumping length sitting still... again excuse position!
People don't wonder at tall riders on ponies, it happens all the time around here [especially on the bratty ponies!]. But, I do find that it's easier to stay on a bigger horse if they start throwing antics. The number of times that little grey brat bucked me off isn't even funny, and his bucks were never very big either, there just was nothing to hang onto [and I can ride through a pretty big buck, had done so maaaaaany times before I got Edward].