There are definitely benefits to being our size! I may not have the leg to ride some of the giant warmbloods effectively but at least I get to ride some pretty cute ponies! I don't think there's necessarily a strict size limit of what you could or could not ride. So much of it depends on their build. As I said, it's actually difficult to ride something that is too small. I tried a pony to buy last week that was 13h but he was an absolute TANK! He felt like a 15.2 horse I ride just closer to the ground! But I worked with a 13 h pony that was narrow as a board and there was nothing to put my leg on. The first time he started his rearing and whirling "trick" I fell off. A pony I have currently in training is 12.2 and not too hard to ride. He's a pretty round guy. As far as if he would be able to physically carry you... he might be, esp if he had a more sturdy build. I know a trainer about your size who does training rides on her small ponies and they are all fine! Plus, you see all those giant cowboys riding around on their tiny cowhorses. So if you follow the 20% weight rule, at your heaviest you could ride something around 700 lbs.
Just wondering, what discipline are you interested in working in?