Have you worked with him on the ground, having him move laterally away from pressure? It isn't actually all that natural for horses to move AWAY from pressure. Most of them will initially move INTO pressure. If you have released the pressure when the horse has moved against it, then he has learned that that is how he gets rid of the annoying thing poking in his side.
Start again , on the ground with horse in halter on leadline and use either your thumb, or a stick (if you need to be further away from biting horse) and push on the various parts of the body (the shoulder to move the front half and hip to move the back half) and don't release the pressure til the horse starts to move away.
At first, reward if you just feel the horse even shift his body away. Then keep increasing the amount of response you need to get before you reward with a release. After a bit, you don't actually touch the horse, but you put your hand near, or very lightly on them, use the "over!" command, and look for them to move away from LESS actual pressure.