Agreed. Temperament and training will be so much more important that the individual breed. Also consider functionality; would you like to have a shorter horse for ease of tacking and mounting or would you like something a little taller for length of stride and probable jumping ease?
Me, personally, I would look for a horse that had a nice, flowing, smooth way of traveling. Nobody wants to spend all their leisure time sitting astride a jackhammer