Hidalgo, this is not a silly question. I think you often come up with good questions, and this is a good one.
I liked all the answers and I was thinking how I could answer different, or add more. Hm m m m. I got it!
To me, a rider is a passenger when they are not in a "feedback" loop with the horse. A feedback loop means that the rider applies a cue, the horse "feeds back" a response which the rider percieves, and thus stops cueing, so horse stops action. It's a dialogue of speaking and listening and it applies to seat, hands, reins, speed, direction , attitude and more. All of these parts of riding are constantly in flux. They can go up or down, more or less, faster/slower, and this is up to the rider to initiate the horse to respond and then the rider to respond to that response and so on . .. a loop of communication.
A passenger is just going along oblivious to the loop, neither speaking to the horse nor listening. They may make a lot of "noise" with their seat and hands by being unable to control them , thus they make inadvertant signals to the hrose, but since they also don't "listen" to the feedback from the horse, the horse soon learns to tune out any noise the passenger makes.
So the rider is oblivious to the hrose and the horse willfully tunes out the rider.