Firstly... how LONG has your SB been under saddle? You say he's just come off the track recently, but how long ago was that? When they're sacked from racing, they go through a stage of losing all the muscle they had as a race horse... the muscle they need to pace/trot is completely different than the muscle needed to carry a rider.
I wouldn't be fussed about the canter right now, sounds like you need to slow the pace down and just go right to the basics of walking and trotting... do not underestimate a 20 minute session working on your walk over a session with some walk and trot that's got lots of moments of not working well. I was once taught that "a good walk leads to a good trot and a good trot leads to a good canter"... as someone who has recently backed and started a SB under saddle (3months now), I completely go by this guide!
I'd suggest you just ride him, not your friend... the more trust and bonding your horse can do with you, the less confused he'll be when some random hops up and asks him to work... that's not his Mummy! And rather than being "clever" I'd say he's going, but you're not Mummy, I don't have to work for you. Honey was the same... she refused to have others mount her cos it "wasn't Mummy"... but was good for them to ride her.
It really sounds like you need to slow the pace down and stop expecting something of a horse that's been under saddle for awhile (sorry if he has)... remember, a horse should take about a year to have a well established walk, trot and canter... and can take a shorter or more lengthier time depending on size and shape. If you'd like some exercises to teach your boy at the walk and then use at the trot, just reply and I'll pop back in here :)
All the best!
♥ Seoul Searchin' for the Lovebug ♥