Comes down to build/strength of the horse vs. your weight/balance. Some horses are not strong and some people are to heavy for all but draft horses. Some people have no sense of balance and are constantly pulling their horse off balance, again a strong horse can overcome this.
Here's an example. My mom has 15hh gelding who has a thin build, probably weighs 850-900 pounds. He's a horse I've never ridden and doubt I ever will. Why? I can reach over his back and pull him off balance with nothing more than my arm. He feels like if I ever actually tried to jump on he would fall over. But I've had a 14.3 mare that felt rock solid under me and didn't hesitate to haul me up the side of a mountain.
Exception to this is young horses, they have to learn how to balance a rider and develop strength to do it.