He is definitely telling you "okay, you're in my space, it's time you get out of it"...This needs to be dealt with, before he does offer a kick, or starts backing in toward you.
You DO need to use your body language to tell him, that HE is actually in your space when he does that, and it is a priviledge, not his right to be near you.
Push him away, using your lead rope, and step toward his flank, so he can see you with his eyes; don't step directly at his rear! Flip the lead at him, and "shoosh" at him. Don't let him back in your space until you are ready, even if you are still treating and grooming the other horses. Face up to him as he walks back up, and ask him to back off while he is still facing you. When you are ready to treat and groom him, then face him and invite him back to you. Groom him, give him his treats, and turn around and leave before he can 'tell' you to leave. Never ***** foot around him, but be confident, and if he starts pinning his ears, and indicating he's going to swing his butt, push him away immediately. Wait a bit, and invite him back into your space, and try again. He needs to learn to respect your space.