True story, happened just today: I was taking a TB stallion out to his pasture for turn-out. He's usually by himself, so he doesn't get to socialize a lot other than over the fence. While I was taking his halter off, a mini snug onto his pasture through the open gate. (They were over-the-fence acquaintances.) I was a bit concerned, but the barn owner told me to let the mini's buddy (also a mini) in as well.
I have never seen so much joy in that stallion about having someone to hang out with. He was prancing around them, jumping in the air, kicking - but never ever got even close to those two minis. He was herding them around for about 10 minutes, occasionally running over to the gate to me, then back to them. Then everything settled and communal grazing commenced. I gave them three flakes of hay (one for each) and again, no dispute.
Now this is an exceptionally kind stallion as far as stallions are concerned, but even with the size difference and with a lot of running and playing and horsing around, there was never a "Oh, that was close!" moment. I guess whether your guys are safe will depend very much on the individuals in question, but there isn't any reason to assume a priori that this is a dangerous situation.