In Wyoming we are a "fence out" state. That means if you don't want livestock on your property you are responsible for securing your property lines. I keep my fences in very good shape because I want my livestock on my good pastures. I also don't want anything from the neighbors getting in and eating my grasses or breeding my animals.
I would treat your situation the same. I would install an impressive second fence to secure my animals. Even if your area's laws are not the exact same, but taking steps to protect your livestock, you would be in a great position if you ever have to file a complaint on the idjit neighbor who keeps a stallion in a single fenced enclosure.