It might be that as you are reaching over you could be bumping him in his off eye, and you wouldn't be able to see it. Do you have your hand up and over his neck (like his neck under your armpit?)
Once you work on getting him to lower his neck, then the rest will be easier.
I am one of those who gives plenty of treats. I do so before and after bridling.
If you are rushing , due to him being evasive, you might be bumping his teeth and only making it worse.
I don't have a lot of help except to work on the "lower your head" , and do this sepreat from bridling; with plenty of pets and a treat or two.
And take your time and stay calm if you can.
Others will have some better advice, perhaps.