My guy is exactly the same way, tough to move off the leg. Lots and lots of transitions, and carry a crop. I ask lightly, then "tell" with leg and crop. We have to work on this at the start of every single ride. He was allowed to be lazy and tune out for several years so this is a looong work in progress...
It also sounds like your guy is bored working on the flat and at lower fences, leading to him getting sloppy. Try incorporating lots and lots of different things, chaning it up every few strides to keep him on his toes. Do you do lateral work? Changes of direction, changes of tempo, etc... Keep your warm up exciting for him, and he should be paying more attention to you.
I do feel your pain, a lot of this is easier said than done!
I've spent most of my life riding horses. The rest I've just wasted.