I agree, I think you'd be fine as long as your ponies are stout enough to take up your leg. (Obviously if you are long-legged and your leg is hanging down way below their bellies, you'll have a harder time and look too big.) Just for comparison, I'm 5'3" and this pony pictured, that I was training, was only 12hh, and even though I was a tad big, she took up my leg well enough that the overall effect wasn't bad at all. And she had no trouble carrying me, which is more important, I think, even than what you look like. (Please pardon the bad posture, I was 4 months pregnant and my back was killing me that day.)
"Keep a leg on each side and your mind in the middle"