I wouldn't say I'm anxious about loading horses myself. I'd call it a healthy desire not to get hurt or killed. I will go in a trailer if I must, but have trained my horse and myself such that it isn't necessary. I stand to the driver's side corner, lead rope in hand and tell my boy to load up. As he does, I toss the lead over his back and follow him in a step to latch the divider in my 2H slant. Then I shut the door and walk around the outside to his head and clip the quick release trailer tie to his halter. To unload, I unclip his halter, open the door and unlatch and open the divider. He'll either turn around and unload himself, or I will tell him to 'whoa,' go into the trailer (on the left, to his head, avoiding his butt) and back him out.
The part I hate is unlatching the divider. If he chose that moment to fly backwards, I would either be run over by his rear or hit by that thing. I don't like it a bit. Maybe I'll see if I can unlatch that from a window... hmmm.
Your anxiety isn't abnormal- it's your brain trying to keep you safe. Listen to it. I've seen people and horses injured too often going in and out of trailers, and the only good way to avoid it is to do lots of practice so it's low-stress and everyone knows the (safe) routine.