When he does this, is it usually when you are just walking past or does he ever give you the stink eye whenever you go to catch him? If he only ever does it when you happen to be walking past, then he may be testing the water for more aggressive actions. If you just sort of ignore it and continue on your way, he may have begun to believe that he was winning those very small situations.
Whenever he pinned them at me, I'd drive him off rather aggressively. You don't need to hit him to do it, just raise your arms above your head to make yourself "big", and give a loud holler. If he doesn't move off quickly, clap your hands together above your head, continue with the loud noise, and move toward him (but stay out of kicking range). That will usually drive off all but the most aggressive of horses. If that doesn't work, start carrying a whip with you and use that to drive him off.
Do this each time he shows you a "nasty face". If he comes back toward you later on with a nice look on his face, then go ahead and give him a scratch in his favorite spot but if he starts looking nasty again, drive him away.
He's making an attempt to prove himself the alpha over you. While it's relatively minor actions right now, if it's allowed to continue it will likely get worse and may progress to biting or charging.