I'm 5.5 and I sometimes ride a 14.1hh grey mare at my riding school. Now, I wouldn't CHOOSE to buy that size a horse for me, if I could help it, but she's an excellent pony, despite being small, and carries people around 5.8 without a problem. I would rather buy a 15hh, 16hh horse horse which I'm comfortable on and can carry me without a problem than go out and buy a 16.3-17hh horse which is too strong for me and that I can't control. :)
You two look great! Perfect size. If you're comfortable on him, then I wouldn't worry that he's the smallest horse in the circuit!
The art of keeping a horse between YOU and the GROUND!