Sadly, it's not about how well trained the stallion is or how good the rider is, someone will be out there on a mare they can't control, and it will be your fault. I have shown a number of stallions, and the first show I competed in, we entered an open gender class. Some older man was up on a mare who started squealing and throwing a fit when she saw my boy, although he was perfectly well behaved, and I kept him as far away from her as I could, a woman with the man absolutely yelled at me up and down about how I shouldn't have a stallion I couldn't control, though at no point did he take a wrong step, nicker at her, or even crane his neck towards her. The man could not control his mare around a stallion, but it is still caused by the stallion. I have since only taken stallions out with experienced riders I know, and in stallion only classes (or one horse in the ring).
The fact of the matter is, if something goes wrong, the stallion and his rider will always be blamed. If your main goal is to have a casual riding horse you can take on trail rides and events, it would probably be in your best interest to geld him. Some mares really do act nuts when they see a stallion, and you'd really hate for something to happen if a young girl is up on a mare that causes a problem she can't handle and winds up injured.