Get what you are comfortable with. Some people just prefer taller horses while others prefer shorter horses. You could go smaller just fine, though.
My husband is 6'4 and he rides a 15.3 hand draft cross and a 16.1 TB. Since the draft cross is wider he actually looks bigger than the TB in pictures. I'm 5'8-5'9 and ride stocky short guys from 13.3 on up. The 16.1 TB is the max height I like to be on and I only ride her because she is so good. Other 16 or so hand horses I'm not as comfortable on.