You will do just fine as long as YOU feel comfortable with it. Personally, I love ponies, and some may say I look silly on them, but I just poke fun at myself for it and make light of it. It makes me feel better to acknowledge it before anybody else has the chance to! But that's just my personality, if it makes you feel better to ignore it you probably can and chances are nobody will bother you about it. It would take a pretty insecure person to make fun of you or anything for how you look on a horse that is plenty able to support your weight.
However, if you are looking to advance and compete at higher levels, there are certain disciplines where you might encounter some problems. But at lower levels at any discipline, you shouldn't encounter any problems, to my knowledge.