Dreamrideredc, you mentioned that there were other horses in the pasture with yours and should you be concerned or what do you do?
I can only tell you what I experienced. T , before I purchased her, was kept with anywhere from 15 to 25 or so horses in a pasture. She was the "boss" mare. Interestingly enough , she would pin her ears at another horse if they approached us and "kept" me all to herself. At least that's how it felt to me.
You need to be careful with a herd of horses, as they are constantly communicating with each other and trying to with us. If we miss a warning message one horse gives another, we could find ourselves in the escape path of the "offending" horse and could be injured.
While I made out fine with T and the herd, I also kept my distance to the herd and T would come to me. If any followed, she warned them. I was not "among" or "surrounded" by the entire herd.
I learnt so much from a distance watching how they communicated with each other and admired how T was a fair but very firm leader. Loved how the geldings tolerated with "unclely" patience the yearlings. So much happens but only if you are looking and wanting to learn.
So back to your OP, yes I think spending relaxed time with your horse is very important to your relationship. But you must also be safe.