Exactly what brightspot says.
It's not that you CAN'T let him hand-graze, it's that YOU have to decide when he can and when he can't, not him. My gelding is a pig and a little food-aggressive. So I make a point of leading him around the yard and not letting him eat grass. I want to be able to take him outside, tie him to a hitching post and brush/wash him without him thinking that he'll be able to eat everytime he sees grass. Otherwise, he is constantly yanking the lead rope out of my hands to grab some. I started by using a lead rope with a chain and putting that under his chin so I have more control. You will never out-pull a horse so you have to out-wit them!
If you let him eat whenever he feels like it, and never tell him no, this horse will just take advantage of you every time. They're like children. So he has to learn that there are times when he will not be allowed to eat. Make a point of it. Use a cue, or set goals (I will not let him stop and eat until I reach turnout and he quietly waits until I remove his halter/unclip the lead rope). You have to be assertive with horses. He won't see you as the bad guy, no more than kids resent adults who have rules and structure, as long as they're fair.