Those little pieces of gravel have to Be removed from the whiteline. Some of the tinier pieces can manage to work their way UP into the hoof, sometimes traveling to the coronary band and coming out thru the skin or the hoof wall. Sometimes they abscess, sometimes not.
When folks post pictures of a hole in the top side of a hoof wall and don’t remember the horse having a painful abscess, the odds are good it was a “gravel” coming out thru the hoof (that originally entered thru the whiteline) in the most non-invasive way possible.
This is an old idea which is incorrect. Unless there is already separation the entire height of the wall, when it would be possible, but extremely rare still, IME. It would only be possible if gravel continued to be pounded in from below, and if the primary problem - severe separation - wasn't already making the horse lame, so much gravel in the hoof capsule very likely would. No, gravel cannot work it's way into hoof wall.
Abscesses aren't always painful(well, that's obvious to us anyway). They don't always make a horse lame. Especially if they're close to the coronary border, or in heels or frog, where the tissue is softer. Abscesses that erupt on the coronary border or top of the hoof generally start there, from a bash to the area or such. Unless there is major, chronic damage to the foot and the abscess at the coronary may be just one 'symptom' of the real prob.
Back to OP... So, aside from them causing blunt farriery tools if you can't get them out, I wouldn't worry too much about bits of gravel etc getting in, but I do usually dig out 'seedy' in a way that doesn't leave any tight holes, if possible. I will often rasp the wall in more of a 'roll' there, to keep it open. That means there's less able to get stuck in there, it's more likely to stay open to the air. BUT in the 'real world' - & I used to live in 'seedy central', where horses were in soft, often muddy ground for most of the year - it often happens that regardless how it's cut, holes can be filled up with crud. And yes, that's more of a seedy risk. So, yes, ideally, hole should be open to the air, but I'd much rather it 'plugged up' with stuff that reduces further harm, than manure & mud, if that's the choice.
You can use clay - I don't know what 'hoof clay' specifically is, but use 'Tuff Rock' clay poultice for all manner of things(good minor wound dressing to keep flies off & aid healing btw) and for hooves, I mix a little copper sulfate with it. Walkin's right, that it doesn't stay in reliably though, as it crumbles as it dries. But you really should dig it out regularly & treat infection topically, if there are any dregs left.
Another option, esp in 'difficult'(muddy, cruddy) environs is beeswax. It forms a mild hydrogen peroxide as it breaks down, so is a deterrent for further infection. Raw honey does, even more so. It sticks in well and it is slightly flexible, so doesn't crumble out when the hoof wall flexes. I make a concoction of beeswax, with a bit of copper sulfate and tea-tree oil in it for 'seedy plugs'. It's still important, if there is any infection at all underneath, to clean out & treat topically, regularly, and I usually advise people do so at least every 2-3 days to begin, then once a week thereafter. But I've come back after 5 weeks to clients and commended them on diligent care of the seedy hole, only for them to exclaim 'oh is the beeswax you put in still there??' So it's pretty effective at staying put!