I hope that "No Trespassing" signs will help to keep unwanted visitors from your cute little colt! He's such a darling, no wonder that everyone wants to pet him. :)
Regarding unwanted visitors, I have several stories of my own.
I used to work at a lesson/trail ride barn that also had a breeding program. Lots of horses of all ages in the pastures and the riding arena, lots of people around every day. Of course, there always were those who wanted to seat their kids on a foal or tried reaching over the fences to pet colts or moody mares, but two cases stand out. First, I once had to excort a couple out of a paddock where they had crawled under a huge electric fence and were preparing to have a romantic picnic. If that's not enough, they said that they just wanted to relax in the presence of the "cute, black horse" they had seen grazing in the far end of the paddock. The horse in question was a breeding stallion, and very protective of his territory, especially against stangers. They were honestly lucky that he hadn't spotted them yet.
On a different day, I went in the barn to feed the horses...just to find a woman and two small kids in the stallion section of the barn! They were hanging out by the stall of a particularly flighty Arab stallion and, as I ran to them to tell how dangerous it is and excort them out...ANOTHER KID CRAWLED OUT OF THE STALL. The mother had just let her kid to crawl in a stall to pet the cute, gray horsey that could have easily killed him! She was completely clueless and regarded me as rude for not letting them stay in the barn not a minute longer, even after I explained how extremely dangerous the situation was!
I've also experienced that people get kind of offended when you explain that horses can cause harm, as if it meant that the horses in question are somehow "not right" for being large and potentially dangerous animals, and as all horses should typically be nice and hugabble for anyone who'd want to touch them.
Regarding my own horse, I dislike when strangers want to approach and pet/feed him. He enjoys company, but can quickly become pushy and start biting if is given treats, as he is a very dominant gelding with a stud-like behavior and needs assertive handling to be nice to be around. Also, non-horsey people are often completely clueless regarding what (or how much!) a horse can eat and I'd hate if someone gave him something he shouldn't have.
I'm moving him to a larger and more public barn in a couple of weeks, and I'm thinking of putting a sign on his stall door, something along the lines of "Please, don't give me any treats." or alike.
I have come a long way, to surrender my shadow to the shadow of my horse.