I'm 5'6" and my gang ranges from 14.1 to 16.3. I'm comfortable on any of them. My shortest is a tank & has ample barrel to take up my leg. I'd not be as comfortable on him if he was the same height and narrow however. My ideal height is 15.2 - 15.3.
The shortest I ride on a regular basis is a student's 13hh Haflinger and in my younger years I rode an 17.3 jumper gelding for a friend - he was a bit too tall for my taste. If I can't mount from the ground, that's too much.
Life is like a camera. Focus on what's important, Capture the good times, Develop from the negatives and if things don't work out, Take another shot.