I have a feeling that your horse is not respecting your space and is all over you when you lead him around. I'm betting that he has an in your pocket personality and that is what is making you nervous, as it should. He/She needs to learn to respect your space.
You need to start working him from the ground. He needs to learn to respect your space. Start by leading him, stop right in front of him, quickly, he should stop. Hold a crop so that you can use the handle as a preasure (don't hit him with the crop unless, us the handle to push him away from you). He should stop whenever you stop. If you walk towards him, he should move away. It takes "agressiveness", when I say that I don't mean "Meanness" like you don't have to hit him, well, you might have to use your leadrope to get him to move away from you, but he needs to always be focused on you and your next move. I think that's your issue, he's pushing you around. I don't think fear of horses is a bad thing, I think everyone should have some fear and some respect, they are huge animals.
You should be able to go into his pen with him.