I know that Minis can be notoriously difficult to get in foal, so I might lean toward the mares not being bred at exactly the correct time, rather than it be the stallion.
Some stallions with very small testicles, do have a poor reproductive outlook too. Could that be his problem, if the fault lies with him? Did his previous owner try to breed him at all? At five, and if they had mares, I'd probably wonder why he wouldn't have been bred. Surely he wasn't in training for five years?
I suppose you could have his semen tested, if in fact the mares he bred, had no trouble getting in foal previously.