I've never heard that rougher or harder ground improves a horses hooves. I always thought hard rocky ground increases the chances of cracks, bruises and abscesses.
I have found feed quality has the greatest impact on my horses feet. My summer pasture supplies a better mix of nutrients and minerals than feeding hay does in the winter (they are on a healthy mixed pasture). In winter, I supplement their hay with concentrates and minerals. If it is very rainy, standing in mud seems to have a negative impact on their hooves, but I can't see how the hardness of the ground toughens up their feet. Their feet also grow much faster in the summer, requiring more frequent trimming, but they don't seem to soften.
It sounds like you are lucky that your horses have good feet. Moving them to grass shouldn't change that, so long that they don't founder or anything.
Maybe I am I'll informed and missing something?