It is not just about how the pony/horse is built but also about how well the rider rides.
I have ridden ponies out Fox Hunting for a busy full day. (5 hours) The smallest was 13.1, she was as stocky little mare but my weight of 145 plus saddle and hunting wear would have brought the weight up to at least 165 - 175. She would have weighed in at around 500. It never did her any harm and she loved it.
I have had horses get scalded backs from riders, who were certainly not to heavy weight wise but because they sat heavy it would blister their backs.