Hi & welcome!
Glad I found the thread. I have recently gotten a four year old paint - love him!! - who has always been barefoot on rocky trail and in hard ground pasture with his herd growing up.
As a 4yo, his feet will only be just beginning to mature in the caudal region(back, digital cushions), assuming the conditions are good for him to do so. It does make a difference, the extra weight of a rider on their back too(try walking on gravel barefoot yourself, then carry something heavy & do it...).
Diet, environment, hoof infections, amount of exercise... all make a difference to hoof health. Great that you've found this thread - & hopefully my links - helpful, because if you start now & give him what he needs to develop great feet, he should have a long lifetime of soundness!
That's not to say he may not still sometimes need protection, and if he's not comfortable, not only is he potentially in danger of direct injury - stone bruises, etc - but he won't be moving properly & can risk other damage so doing as well. So yes, it's a good move, and you're not 'babying' him to provide it when necessary. Boots are generally a great option if/when needed.