Hmm... a question with lots of variability in the response it'll get!
When I selected my trainer, I was lucky enough to have watched her ride at several "free auditing" horse events. I approached her after she had ridden and just kind of struck up a general conversation to see if I even liked her personality She was very down to earth, and very thorough and kind in the way she explained things. I went home, did some research and found out that she was not only a great rider with a great personality, she has a record of coaching beginner riders and up through successful Grand Prix show careers, has trained several winning Grand Prix horses from the ground up... All of this on top of her own USEF and USDF gold medals, board positions and many other "official" credentials. I've been riding with her for almost 2 years now and feel that my horse and I have come great strides in connecting with one another and learning together. I pay a pretty good chunk of money to work with her, but it is absolutely worth it to me. I'd rather do two lessons with her than 4 lessons with someone less experienced and successful. I wish I could train with her twice a week!
I know that not everybody is willing to shell out money for a trainer, and everyone's opinions of "reasonable" rates vary, too. What are your opinions? Do you look for credentials? Do you look for experience? Do you look at how expensive they are (as in "They cost $200 per lesson, so they must be better than the lady who charges $25...")?
I guess I feel that you get what you pay for in most instances... I've witnessed people who call themselves "trainers" charge someone $20 (cheap, I think...) to spend 3 hours essentially "ruining" the person and their horse. However, I've also watched someone training a horse and rider, charging $20 per lesson, and the student and horse walked out of the arena with a better understanding of one another and how to connect.
The difference here being that the first trainer had no real "credit" either in their own riding successes (other than keeping their butt in the saddle when the horses acted up), nor did they have any credits in training anyone else's horses to successfully connect with their riders. The second trainer had some successes in the area of expertise the student was seeking, and a few dedicated students who were showing improvement in the show ring in the the discipline of the student's choice.
I dunno... interested to see what all variables people consider!