I am five foot five and 104 pounds. There are quite a few horses in my pasture, but the main ones I ride are High Five, 15hh, and Vega, 16.3hh. Oddly enough, I fit on both of them perfectly. High Five is a very stocky, foundation built Quarter Horse, and Vega is a lanky Appendix.
Honestly, as long as you aren't overpowering the horse/pony, you should be fine. You two look fine together, but if you are worried, consider getting a Quarter Horse. Typically one that is 14 hands or so can handle a pretty decent amount of weight up there as long as they are built well.
Red Money Maker (Red) - 2004 Sorrel QH Gelding