Like others have said, they can pull them off.
Every time I have to search a pasture or paddock for a bell boot I am thankful the bell boot was there to be pulled off, instead of the shoe!
It you go too much smaller (not touching the ground) like was suggested you will not be providing as much protection to your shoe.
I suggest you keep a spare set around to be able to replace the bell boot when the missing boot is hiding out there some where. They really can fly pretty far if they come off during very active horse fun.