I really have no helpful advice just commiseration: Lacey used to be the exact same way. She knew what "whoa" (or in our case, "ho") meant at a walk but any faster than that, allll was forgotten.
In her case though, it was mostly due to nervousness and basically "leaving" the situation mentally because she couldn't handle what I was asking for (too much "pressure"). As we got to know each other, and she learned that I was not going to ask for "too much", she calmed down quite a bit and became more "present" all the time.
I still can't get any sort of "whoa" directly from the canter and she's certainly not reliable about it at the trot, but she will reliably drop down a gait with "easy" so I just drop down through the gaits until I get her to a walk where she will stop. Most of the time, a canter-trot transition+whoa will equal a full stop but sometimes that trotting is just SO MUCH FUN.
In her case, at least, I'm sure it's largely due to her sight. Over the summer, when I had Lady, I realized just how much "normal" horses go off our body language while round penning/lunging and also how little Lacey uses my more subtle body language during those same times. Prior to Lady, Lacey has been my only round penning/lunging experience so I had no idea! Haha
While I've been yammering, two things came to mind: what is HIS body language saying? Is he excited or panicked? Is he running around "dead" - like he's not accepting external input? Or is he playful seeming? You've said what he does but what does he LOOK like while doing those things?
I would hazard that he's overwhelmed/"dead"... Maybe try toning down your body language and just see what happens?
I know sometimes I can get "harsh" with my body language and I don't even realize until I wonder why on earth Lacey is acting so nervous/tuned out with me! Then I tone it down and she returns to her usual self.
Anyway, lots of random thoughts here... sorry about that.