You don't have to let her ride your horse if you don't want to. ...well, what I mean is, don't do it if you feel obligated or pressured to let her ride your horse.
I pretty much was at the barn six days a week, so it would have been too much if someone else had also worked my horse. However, I did experience this when I had to find someone to lease her. What I did was have the potential ladies take several lessons on my horse with our trainer so we could see how well they clicked. It really helped all of us involved and now my horse is being completely spoiled rotten by two wonderful ladies and is teaching them to improve their riding.