Perhaps he doesn't really understand? You could try doing some ground work and ask him to back up from the ground with gentle pressure on the reins. If he ignores it, tap him on the chest with a crop until he takes at least one step backwards. Reward him, and practice this regularly. He will soon learn that gentle pressure on the reins is asking him to back up.
When I bestride him, I soar, I am a hawk: he trots the air; the earth sings when he touches it; the basest horn of his hoof is more musical than the pipe of Hermes. ~William Shakespeare