I'm a hoofcare practitioner. I agree that it's normal, but that's not to say that it *should* happen, is good, is unavoidable, etc. I don't like to see a horse uncomfortable on his feet and it's not productive anyway. I would be looking at the causes and remedies for this. Studying Pete Ramey hoof care heals founder in horse’s navicular disease farrier might help you there too.
Many farriers rasp into the sole, especially at the toe, where many horses - esp. Those that have been in shoes for some time - already have very thin sole, due to this practice, often in conjunction with leaving the heels too high. Apart from this problem, gimpiness when bare often seems to be from other parts of the foot, rather than 'lack of callous' on the sole.
If a horse is left with walls being substantially(more than a few mm) longer than the walls, this can create pain, as the leverage forces rip the laminae apart, at the top & bottom of the wall.
If the heels are/have been high, removing the heels & frogs from ground contact, &/or the horse is always on softish ground, the heels will be out of condition to take the weight of the horse on at least some surfaces. This tends to also lead to the horse landing toe first, to avoid the heel sensitivity, which leads to further problems...
It's great that you give your horse a little time off from shoes each year. 6 weeks is a very short time tho. It also depends on how he's managed/trimmed in that time, and the rest of the year as to whether he will benefit much from it tho. If he's gimpy without shoes, he won't be using himself properly, or keen to exercise when he's not made to, so it won't achieve a lot. Assuming he's well trimmed & not lame because of the way your farrier has trimmed him, I'd be getting him some boots for these occasions, so you can exercise him to the fullest without him being uncomfortable.