I would look at conformation more than height (depending on your own size, of course). I think a shorter horse with good conformation, good bone, and good feet would likely hold up just fine. A taller horse with the same characteristics should be fine for ranch work, too.
I had an uncle who used Arabian and Arabian crosses for heavy ranch work. Thoses horses covered a lot of miles, and even the small ones held up just fine. My uncle was quite slim, though. He also had some tall Thoroughbreds (and TB-Arabians). The tall horses were certainly agile enough for ranch work, and they were great for jumping, too! He didn't have a QH on the place.
So I wouldn't sweat height too much. If you can watch your prospective horse playing in the pasture, that will give you an idea of how agile it is.