I am with those who said it's incredibly rude and dangerous of everyone to go running off. Even if one stays, your horse is going to want to go with the larger group.
My trainer tells me that I can handle my sensitive, powerful, green broke horse, but she also doesn't push me. Most of my rides on him have been just walks around the arena.
Please don't feel bad that your kids seem to be able to control him better than you. It's quite possible they haven't pushed him or tried to really tell him what to do... they may be just giving in and making it look like it was their idea.
It's also possible this horse is simply not used to adult riders, and it may take time for you to really "click" with him.