Sounds to me like your horse is set into its routine. Sometimes, certain horses will use the "routine" against us by taking our idea and running with it.
First I would practice haltering etiquette without the food factor. Teach him a cue that means it's all right to leave. Leave he lead rope around his neck when you take the halter off so that you can pull him back to you when he tries to leave. Make him put his head in a submitful position ( neck lowered, head and nose tilted toward you and within close reach). As soon as he holds that position and relaxes (and relaxing is important) give your cue and let him do as he pleases. My cue to my horse is a few pats on his chest or shoulder. You could do that or use a verbal cue. I find that physical cues (body language and signals) are much quicker for horses to learn. Keep practicing asking for your horses head and then releasing and cueing when he behaves and relaxes. Practice it out in the lawn. Let him graze for a little while ask him to lift and give his head to you ( If he pulls and doesn't listen, instead of pulling and trying to haul his head up, just ask first with a gentle pull and if he doesn't respond, bump on the lead rope with increasing pressure till your horse gets the message and raises his head for you. ), and then hold his head and ask him to hold his head respectfully in position for you before you cue his release. Keep practicing until he's real respectful and responsive. Then do the same thing at dinner time. Wait for him to hold position and relax (without his halter on) and then release him.
Another thing that helps, which I do with my horse so that he doesn't learn to get jittery at dinner time, is tie him in his stall and let him stand for a while and relax before letting him eat.
So #1 tie your horse up. #2 then put your feed in the stall. #3 wait for your horse to calm and relax (which could take a while. My horse was so impatient at first, but after a couple consistent lessons of this, he calmed right down)
#4 after he calms down, walk him to his food and remove his halter, reinforcing that he keep his head in position. (I do this with his head right over his food. It teaches really good impulse control) and then #5 when he relaxes and is respectful, cue his release.
This method works great for dinnertime pushers and for horses that pull your arms out of your sockets the minute you hit the lawn. They learn that they can eat and enjoy themselves, but they have to wait for your permission. I also can't stress enough that you reinforce relaxing! ( softening of muscles, lowering of neck, slow blinking, licking of lips, cocking a leg, taking a deep, slow sigh...) make your horse think that the food is available only if he's relaxed. And once he learns this, occasionally, while he's eating, interupt him and ask him to give you his head. Relax. Then release him.
This has worked miracles for me. No more pacing, no more tense pushyness. My horse doesn't zone me out whenever there is food involved. He takes his time like it's no big deal.
Now, I also, when my horse is eating, I also like to groom him, scratch him, and just hang out. This teaches my horse that he doesn't have to ditch me to enjoy himself or get a bite to eat!