Try to put the fact that he's 4 to the back of your mind. I also started with a 4 year old. On age alone, my brain said I should expect him to do stupid things, but he never did. He ended up impressing me more everyday. I quickly went from worrying about him to trusting him completely and he's since returned the favor. If you like this horse, don't overface yourself in terms of showing and training, but do trust him. Sounds like he's already earned it.
You just have to see your distance...you don't have to like it.