I went to AQHYA World one year to do horse judging, and our team watched the show jumping finals. One of the top five was riding a quarter/pony! That things flew over those jumps and he was 1/3 the size of the other horses. They were big jumps too (I really can't estimate the exact height...but world final quality)...I don't think height matter so much if your horse has the talent, ability, and the mind to do it.
The weight of the rider will effect the horse some, but as long as you are physically in pretty good shape and the horse isn't tiny, then it shouldn't matter that much. How well you balance and move with the horse matters more. If your weight is in the wrong place, at the wrong time, you can throw off your horse's balance and make it harder for him to perform. If you take lessons, etc, and learn where you need to be at all times, and learn to synchronize with your horse, your performance will get better and better.