It probably is banging, but it could also be banging out of his mouth, don't rip the bit out of their mouth, instead hover it as you pull off and let them remove their head from the bit. Try switching to a rubber bit just to be sure, it may not work at first, but once he figures out that it can't hurt him and you won't hurt him, he will probably get better about it. And familiarizing yourself with that spot you can stick your fingers in can be helpful, I always do it so their mouth is open before the bit even gets there, but you have to stand beside your horse facing the direction he is, and be able to bridle that way. I see so many people who have had horses for years, not know the proper way to bridle and stand facing them to bridle. Yeah it works, until a horse that is headshy, sneaky, scared, young, green, etc. comes by. Really learn to bridle with the noseband, headstall, throat latch, gathered in your right hand, and the bit in your left, which positions it perfectly to stick a finger in the side of their mouth, if you practice this, you should be able to bridle any horse.
Troubled TB ~"A thorn by any other name will prick just as deep." @-'--,---