I'm 6'3 160 lbs and the average horse will carry me just fine and the same should hold true for yourself. You can ride a smaller horse as long as your balanced. Often weight isn't the issue, it is how the rider and saddle distribute the weight. If the saddle doesn't fit or the rider rides unbalance then too much weight is put in one spot and causes issues. Here are some clues to let you know there is a problem; White marks from saddle, lameness, horse showing obvious discomfort (throwing head, bucking, etc).