When I first started riding I spent a good part of that year on a sweet fleabitten grey pony named David. And I was at least 5'10" at the time (I was 11!). I did look a little silly, but he was pretty stout and on the tall side for a pony, 14 something I think... I never had anyone complain about my being too tall, and I even took him to a hunter pace - we kept up with the "big horses" fine :)
After that I leased the most lovely thoroughbred mare, Dolly. She was 15.3 hands but she was dainty and pretty skinny. So even though she was bigger, I was still 5'11", and I always felt kind of big for her. I guess that sort of proves it really depends on the build of the horse!