I used to let friends ride my TB, he was a bit hot but I knew in the arena he had so much attitude he would just refuse to move for a beginner, he thought he was way above them. You'd have to be pretty decent rider to get more than a plodding trot from him.
When I had my mare, she was 4/5 at the time, I would not really let anyone else ride her because I thought it could be unsafe. The only people I ever let ride her was my instructor when I didn't have the time to work her, and the people trying her out for sale, and I certainly warned them she was young and I would not guarantee her behaviour or their safety, but I rode her first. If friends pestered me a lot I'd suggest we go to the local riding school and all go on a paid trail ride together, or if I was in the position, borrow a friend's quiet horse.
It's a bit of a tricky situation because while other horse people, like people on here, understand why owners wouldn't let their non-horsey friends ride horses, those non-horse friends don't understand. They think horse riding is easy and fun, and they see you ride around on your horse and everything goes perfectly and it looks like you are doing nothing but sitting there. They don't understand when you say no, and think you are being selfish or judgmental or arrogant or something.
It helps to have big horses though. Most of the horses I've owned were over 16 hands and when people meet them they are usually too intimidated to want to ride. Or I say something like "if you can get up you can ride" and they never can.