My hubby had our horse on a leadline and was walking him. Technically we were doing a pony ride. Any riding we do is closely supervised and pretty controlled. Still, as you say, horses can be horses.
It goes back to "buy a horse that is in your experience level" I think I will talk to the BM, it's just going to be hard not to sound petty and snappish.