It also really, really depends on what level of riding you want to do.
A horse that is cow hocked, downhill, short neck will likely have no problem whatsoever at the lower levels. But ask that same horse to jump higher, or have more collection and he'll find it next to impossible.
So you need to determine what you can/can't live with. For me, I wouldn't touch a swaybacked horse. That's just asking for problems. More and more, I'm also looking for a horse with a good sacroillian support, because I've had problems from previous horses. Quite a few horses are downhill (especially thoroughbreds. 95% of TBs are all built downhill), and that wouldn't automatically deter me-rather, they need to be able to MOVE uphill and carry themselves.
Conformation also doesn't dictate everything. We have a horse at my barn who is competing Prix St George this year. Ugliest thing you've ever seen. 17.2hh, downhill, swaybacked, HUGE neck and weak loin connection. But, while his movement isn't the flashiest, he's honest, has a fantastic mind and is traineable.