Absolutely, do not do anything that further irritates your back! I broke three of my vertebrae in a fall and rest was the best (and only..well, a horrendous back brace helped, too) medicine for your back. It's such a vital part of our bodies, we need to respect it.
Like others have said, groom, clean and watch. You don't want to do anything that will cause further damage and lengthen your time out of the saddle!