It is my belief that you should always, ALWAYS go with your gut feeling. There is a reason that no one can ever fully explain why we get these feelings.
Psych nerd moment:
Our brains have stored much more than we realize. Either it's decided that certain "files" of information aren't important enough, or there just aren't the connections made, so we can't consciously access this information. Hence, sometimes we get a "gut" feeling but can't explain why. It's usually a good idea to go with your gut for that reason.
Of course, you could also take the religious approach to this:
The spirit of discernment has moved you ;)
And finally, as someone else suggested, it may just ge nerves, but sometimes that's a good reason, particularly with horses, because your nerves could make an ok situation into a risky one as the horse(s) picks up your vibe.
Either way, to me it doesn't sound like it's worth taking the risk. There will be other opportunities (I'd start with calling the trainer and explaining how you feel and asking what other opportunities there may be), or just go and stay in the arena and explain why you need to do so (I don't trust that my horse and I can do this outside the arena). But don't risk something you don't feel comfortable about, especially when you're trying to learn and have a good experience.