Welcome to the forum, smezera!
This is herd-sour behavior. This is one of list of horse-addictions that we fight with our horses about pretty much their entire lives. I say "Addiction" bc you can train them to behave, but a new owner can let the same horse slip back into the old behavior really easily.
Since you can ride him, he isn't a weanling or a yearling--you couldn't allow them to fuss alone bc they would injure themselves.
I say, let him call, fuss, fume and run. Ignore it. My 5 yo QH wasn't trained well enough for us to take him with the other 2 members of the herd last month. We had a program with children and I didn't want any accidents. We took out the other horses, made sure that all of the gates were secured, checked his water/food, and left him. He was there, waiting, when we came back. He recognized the trailer and followed the fence until we pulled in and unloaded.
He didn't harm himself.
We also left him in a pen when we took him out to SD to trail ride last year. He JUST wasn't quite ready for the trails--squeeled and tried to take off with me, so I "grounded him." So, he spent every day waiting for us to return. Still didn't hurt himself.