It sounds like he is not comfortable enough in his stall to relax and eat.
What is your routine and can you change it up? My horse dropped a lot of weight in a barn where the horses were kept outside all the time except for being brought in for an hour to eat. She was not comfortable in the barn at all, and was anxious about the big rush when the handlers would let all the horses back out again.
Another routine that often does not work for anxious horses is to feed the grain in the morning before going out because they are anticipating going outside.
There are a bunch of different ways to make it work, depending on your situation. If your horse is outside all the time, is there a way to fence of a smaller area or pen where your horse can be separated from the others while eating grain but yet see the other horses and not get anxious? Then you could leave him separated for a few hours or overnight in order to give him plenty of time to eat.
What worked for my horse was to be fed all her grain at night and to be left in with a good buddy in a stall nearby so they could relax together. Some hard keepers need a substantial amount of time separated from other horses where they can relax and eat. But it won't do any good just to put him in the barn to eat if he is fearful of being inside.