Personally, I would never strap my child on a horse or pony. If they aren't coordinated enough to keep themselves on the horse, they aren't ready to ride and as someone else said, they "can't" fall off, even if the horse spooks, falls, etc.
I have ridden since before I was born and I rode bareback first on a big buckskin mare from around 2 1/2 yrs. up. My sister and I developed really good instincts for staying with a horse. It's hard to find a "pony" saddle that fits a horse.
I don't know how you feel about riding double but there are "buddy" saddles that strap on behind your saddle and fit a small child.