I just had a similar experience on my last trail ride. I had my green 3 year old out on a rural road and a car passed by very quickly and very close which upset her. She calmed down quickly though and she'll get over it, but I did complain about it to my husband, haha.
But, what I really wanted to say - if this is a common problem where you live, try to think of yourself as an ambassador of all trail riders. Drivers may honestly not know a thing about horses, how to pass safey, or that it could be dangerous to pass. It should be common sense to pass with caution to everyone, but not everyone has common sense.
So, as the ambassador, when you see a car approaching, move to the side, stop, smile and wave at the car. Instead of being "that jerk horse rider that takes up the road", be the friendly horse rider that is courteous, cautious, and friendly. People will interact with those that project friendliness, and be likely to slow down, and perhaps even say hi. If they do, thank them for slowing down. Resist being mad with them. People do not like to interact with angry glares, shaking fists or obvious signs of irritation, and will likely speed to get away from you.
And I get you don't mean you would actually throw rocks, but if you did, first off they would know it was you and not the horse, but second it would make them irritated at all riders and would not encourage them to pass safely in the future. Your intended point would not be the point they get, they'd just think you were a huge jerk.