I think it is quite common, actually. Not so much that a horse is a 'one person' horse, but that given opportunity, many horses will be lazy or misbehave.
My horse, for instance, can be difficult for other people to get him to canter. I had 2 friends out, one he wouldn't lope at all, the other maybe 4 strides. I got on, just say 'come on', and he lopes beautifully around the field with no resistance. Not because I am the be all and end all of riders, he just knows by now that I won't take any bs.
The dude horses when I worked at a ranch were notorious for blatantly ignoring cues if they did not know their rider.
This is why you hear countless stories of how horses 'change' when they are sold. They are no longer under the watchful eye of 'mom' or 'dad', and they will test you.
So if you are really interested in this horse, I would spend a little more time. The horse is obviously capable of riding well, he just wants to make sure You know what you are doing. :)