Sounds like a serious lack of respect issue. You will probably have to re-start his ground manners as it seems like there are holes in his training. No matter how big he is, he needs to understand that YOU are the boss.
My friend's standardbred was like that and this is how I fixed it. I put the chain over her nose, and when I was leading her, if she pulled away, I would lead her into a tight circle and just keep circling until I had her attention. It is a very long and frustrating process, but sooner or later she got the idea that pulling/barging= circles.
If you don't like the idea of the chain, put his leadrope on, and when he tries to barge ahead, tug on the leadrope over and over and ask him to BACK UP. You just have to be really firm, as you are right, someday he really might hurt someone. Good luck!