I'm 5'8" (5'9" in my MuckBoots) and my horse (pony) is 14.1hh on a tall day. However, she's very wide and most of my height is due to my long torso so we fit together pretty well.
I wish she were a bit taller because it is hard for me to really use my leg effectively with her, but I can't complain.
Just for funnsies, since I think it's hard to imagine (and I like pictures), a picture to demonstrate (excuse all the paint and weirdness happening, hahaha):
I have the easiest time giving cues on horses 15.2+ hands but I've ridden horses as small as 12hh without major issues. For me, it all depends on how the horse is built.