I think you have to look at it in two ways: what is "natural evolution" and what is "genetically modified" aka what we humans have bred into them over generations to suit our needs for work, pleasure, and competition.
For example, over time we have increasingly bred longer legs and necks into Thoroughbreds, Sporthorse-types, etc for use in racing and sports such as Jumping and Hunters. The opposite is true for cow horses, where we continue to breed them relatively short and stocky as that is better suited to the purpose.
With many species, getting larger or smaller goes hand-in-hand with avoiding predators and/or getting better access to food and basic needs for living and reproducing. I believe that is also true of the horse, but once we began domesticating them I think we interrupted natural evolution and thus began our own genetic modifications to suit our own wants and needs, the same as with dogs.