In my opinion, you should be able to get in his space and do what you need to do, but he doesn't need to get in yours. If he does, step at him and jerk the lead rope toward his chest (kind of banging the noseband off his nose) and make him get back! A few steps is good. When you lead him, be aware that he's not crowding you. If he is, do the same thing...make him get back. When you're leading him and you stop, he should stop and back up a step. I make mine follow a good 6 feet behind me. If something scares them and they're right up on you, you're run over.
I have little kids too and they're taught this. Even if he's a big-ole loving cuddly horse, it hurts just the same if he steps on a toe out of meanness or by accident. You don't want those young-uns to get stepped on!