Depends on the class. In Western Pleasure, yes conformation is a factor, and in halter conformation is basically the only factor.
But most likely it's just the fact that the lookers always have owners who know the most about showing. It's sort-of difficult to explain, but one thing leads to another.
Don't expect to be a winner immediately. I didn't place at my first show. Most people don't. Just don't get discouraged.