What makes you think the 'games' you play with your horse are something he/she considers fun? As far as the horse is concerned it's all work, regardless of what the clueless predator on the other side of the round pen thinks!
I do not ask my horse's permission to ride him, just as my supervisor doesn't ask my permission to do work for pay. Oh, no thank you, boss! I'd rather just sit her and play on the interwebz all day!