Ya, it's a drag when you have intolerant neighbours. We have one (no, actually two) of those here. Thankfully, everyone else is of the same opinion of them so if push ever came to shove I would have support. There is nothing you can do about that. The neighbour is in the right, legally speaking -- he's just being an a$$ about it. Chalk it up and remember that when something goes wrong for him. That's what I'm waiting for with the No. 1 "perfect horse owner" on my road. (Her self-characterization, no one else's.)
If your fences are secure and the only reason this happened was because your grandfather was irresponsible/naive about moving the animals, then you need to educate him or tell him he's not to move them EVER.