I would definitely opt for keeping the feet clean rather than letting dirt/manure pack in there and become a breeding ground for disease..
HOWEVER, I did spend 6 very hot and dry months in Wyoming where no one ever cleaned their horses feet. I cleaned out mine anyways but thrush was literally unheard of there, and all the horses were very healthy.
So I suppose it depends on your climate, if your horse is susceptible to thrush, and how comfortable you are letting his feet stay dirty.
I would recommend talking to your farrier about trimming. Let him know that you're concerned about foot health, and he may be willing to talk to you about it, and perhaps let your horse's frog grow longer. Mine does, but he's a believer in 'natural feet'.
"There are two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle." -- Albert Einstein