I agree with Tinyliny & Yogiwick,
My horse is the same, he was always pinning his ears when I would take his food out, and he would pin his ears & act quite irritated while I was messing with him (rugs etc.) while he was eating.
I now know that he just really loves his peace & quiet while he's eating & the ear pinning is actually just him, not aggressive at all...
I had kept him away with a whip a few times while I put his feed in the shelter & made him wait calmly, then he'd calmly proceed to his food with pinned ears haha, so I just let him be, he knows not to come close while I'm waking with the feed bucket & he waits until I place it to go & eat.
So long as he's not invading your space or being aggressive, just place the food & walk away & then go back afterwards to do rugs etc. or do rugs before you feed.