My horse is 16.2 hh and I've fallen off him twice. The first fall happened when he stopped dead in his tracks right before a jump and it was quite bad actually, because I hit my head hard and ended up with a concussion, but the other one was very soft - I came off in trails and just flew in some bushes nice and fast, without any injuries. The falls really depend on how your body reacts - if it stiffens up or if it knows how to relax. One can have very serious injuries falling off even the smallest of ponies, if during the fall the body is tense. Around here, almost all horses can be considered tall (16 hh and up) and people are just used to falling of these heights - I believe, it is not much different than falling from a smaller horse.
I have come a long way, to surrender my shadow to the shadow of my horse.