I have to say, I think you would look "better" on a bigger horse, just because you'd look more balanced. Same with me though, I probably wouldn't have noticed if you hadn't pointed it out.
That being said, who cares what you guys look like if your horse and you are comfortable? I am 5'8, my QH is 14.2. He is exceptionally stocky so he takes up my leg, but I have a long torso so I look like I'm towering above him. I don't care, however. I've had him 11 years and he's what I'm most comfortable riding. We've also never had a problem with showing due to a height difference.
I've spent most of my life riding horses. The rest I've just wasted.