It's a bad idea, but you already know that.
And yes, I've done it, multiple times. Rode with both arms in elbow length casts, and then with both in splints. Also rode with my right ankle in a splint type boot and a temporary pin.
More recently, I rode 4 weeks after total knee replacement. And a week after knee arthroscopy, with stitches still in.
But I'm a risk taking idiot; I am not recommending my practices as good or sound practices.
Managed not to injure myself further, but doesn't mean that the potential to do so wasn't there.
Doesn't make it a good idea.
Really comes down to your judgement and what your tolerance for risk and reinjury is.