Yes, the build of the horse is so much more important than its size/height, although of course you may feel more comfortable on certain sizes of horse. Of course, the size of horse you may be accustomed to or other people may be expecting to see you on varies quite a bit with the type of riding you will do.
I grew up doing eventing, as well as some stints as a working student with dressage or hunter/jumpers barn and now ride at an eventing barn and foxhunt frequently. In these sports, it is often common for even pretty small riders like me to ride large horses. So when my husband (~6 feet tall and 220 pounds without tack) and I started shopping for a horse for him, everyone we know expected us to purchase one for him that was 16+ hands. What we ended up with is a 14.3 hand quarter horse mare who is just SOLID. She requires an extra wide tree for her english saddle and has plenty of bone. Her barrel is so wide that even when fit she takes up my 6 foot tall husband's long legs a lot better than many taller but slab sided horses. Even more importantly her temperament and experience are perfect for what he wants to do, so although he occasionally looks a little out of place on his short, chunky little dun mare he couldn't be happier with her.