You know it really depends on the build of the horse. A lot of the time you can get away with smaller horses if they are thicker. I rode my old trainer's pony for a few years and he was only 14'1hh(I'm 5'5) but I got away with it I think mainly because he was big barreled.
Karen O'Connor rode a pony thru a 3 day event. Teddy was only 14'1hh(the poor guy was put down in May-a injury during his handling).
Unless your feet are dangling way past the barrel of the horse you are riding, or your feet dragging on the ground
I wouldn't worry to much about the size of the pony.