The taller the horse the farther it is to fall and the higher the stirrups for getting up there
I am 5'8" and ride a 15.1 horse. But then I just turned 50 today (happy birthday to me!) so I'd have a little more trouble getting up there if she was taller. My last horse before Vida was a stocky draft pony QH mix that I believe was around 14.3 she carried me for many miles and I never felt silly riding her.