I guess it depends on what else your horse has to eat.
My horses have been exposed to Walnut trees their entire lives - two of them are in their mid-20's, the other two in their late teens. They were on very large acreage, so lots of forage to choose from.
If your horse is on tiny acreage or drylotted and there happens to be any sort of tree in the lot, then I would worry. I would worry about the tree, as well, because horses WILL eat bark out of boredom