If you call everyone and only bring in one horse, they will start to ignore the call.
I suppose if you call everyone & then ignore most of them, or chase them away they will. I tend to take the time to give them all a scratch or something & then take the one I want.
Another concern I have is that someone told me that putting horses in a herd setting makes them herd sour and not as responsive to their humans.
I will always keep my horses in a herd where possible, though at the mo it's only a herd of 2. I don't think it's fair on them to deprive them of a normal social life with other horses. Never had a problem calling them, after they initially learned I was a Good Thing!
If you're not nice to be around, then of course a horse would prefer to stay in the paddock with other horses. It's the one(s) left that 'spit it' when I take someone out. Very 'herd sour' when left behind!