Your body (and your horse's body) needs energy in order to complete basic functions such as food digestion, circulation, etc.
If you rob a horse of its energy, it cannot complete those basic needs. It also has no chance of improving in condition because it will be using its reserves for your own selfish needs.
There is also less padding so there is bigger chance of injury. Also they are more lethargic, and more exposed to feeling blows. Without fatty tissue things like kicks from other horses will be more severe and damaging.
And it's just really inconsiderate.
"Strength is the ability to use a muscle without tension"