I do not think it is too much to ask of a horse, to stand nicely while the others gallop off, but it is certainly something that must be trained for the horse to understand.
I can use my own horse as an example in this. He's 6 years old and I just purchased him in May. I've not gotten the opportunity to ride in a large group of people (most of the riding I do is solo) but based on his personality, he would be the type to "freak out" when the other horses go galloping off. I know at this point in time he is not ready to be made to stand still. I would have to do random exercises with him (circles, serpentines, flexing, etc) to keep his attention on me and not the other horses. But it is something to work toward and something he could eventually handle. But I know right now if he was in that situation, he's want to run with them.
It's all about what you horse has been trained to do and their personality, and how you handle it.
I do find it very rude and disrespectful that ALL the riders in your group would leave you behind, just because you aren't ready to gallop yet. Apparently they've forgotten how scary it can be to RUN on a horse when you aren't ready to do so. Shame on them for not finding a useable working situation for all of you.