Any horse around young children should be something to keep an eye out.
Introduce your kids to the neighbour, and check out their fencing- whether your kids could slip through the fence, it's electric etc. At the end of the day, if your kids get in it is your responsibility. I'm not sure how old they are, but teaching them to stay away is your best option, or to keep them supervised.
^^^This --- 50 times over.
Before the little house 12 feet from my barnyard fence got sold, it was rented out many times; often to folks with small children.
My barnyard is my sick bay for any of my horses.
I did not wait for new renters with children (of any age) to start a conversation. The first moment I saw them out with their children, I called them to the fence, introduced myself, and dove right into graciously telling them why their children, should NOT
1. Climb my fence
2. Feed treats to any of my horses
3. Could not pet them unless I was out there to supervise.
I don't mean to sound harsh but, it is your responsibility as a parent to keep your child in its own yard and its fingers out of a horse's mouth. Regarding the stallion:
Yes, a sturdy fence is needed and if you plan to bring horses onto your property, discussion with the stallion owner regarding an even sturdier fence is needed.
If the fence between you and the stallion is regarded as a "common fence" and you feel the security it offers is minimal, you should step up and offer to help pay for a sturdier fence.
I know that comment will grind the suburban and city folks moving to the country but, that is how things work.
DH and I paid for erection of the fence in the back yard. The perimeter fencing around the old cow pasture was already here and being maintained by another neighbor with cows. When he got rid of his cows, the maintenance fell on us, if we wanted to keep the horses in our pasture.
That is just how things work in the country.