goats basically ignore fences, If there is plenty of food inside the fence they will generally stay in the field. They will eventually bond with the horse and form a herd. You do not want a male goat, trust me on this. Get a gelded male or a female. They really don't need much care at all, maybe hooves trimmed once in a blue moon if ever. Bag of goat wormer , tetnus shot. That's about it. The eat grass and weeds, bit of whole corn. Don't give them horse food. Horse food has copper which is bad for goats. The little bit the horse drops wont hurt them. Give them some shelter like a big dog house up off the ground a bit, they like to sit up off the ground inthe shade where they can look out at things. Goats are very easy and require minimal care. Which is why they are used as companion animals.
The big issue is the fence though. They will ignore it, might be a problem for you.