The colt has actually chased both owners out of his pen, as well as bit them, reared up, and kicked at them.
You said he is aggressive towards you? Has he tried the above with you?
I have to say I have never had to work with a horse THAT aggressive towards people, but here is what I would do (make sure you wear a helmet when working with him). It sounds like you have the right idea with starting him. He must have not been handled and worked with at all which is why he probably treats anyone who comes in his pen like someone to keep away.
I would keep him in the round pen and make him WORK. If he wants to be an piece of asset, then he's going to be working his asset. End of story.
My rule of thumb is if they have the energy to be jerks towards me(other than trying to tell me something like express pain, or fear), then they will have the energy to work hard. Years ago, a gelding I owned chose he would not load in the trailer. There was no reason for it, he was the most wonderful loader in the world(but my mistake was letting him get away with being a pest during loading the previous time). I refused to fight with him, so we got in habit, that if he didn't want to load, we would immediately lunge, and lunge HARD. It took 2 situations like this in a row, for him to remember what his options were. Never had problems after that.
Once he learns about respecting your space with the lunging in the pen, then start asking to halt, or work down to a walk. Get him to start acting like a civilized horse. Work your way to being able to get him to stand for you, even come to you. It might take some time depending on set he is in his ways, but it can be done. I'm sure he is a wonderful horse, but he has no concept on what respect means, and he needs to learn that.
I don't if it's an option, but I would love to see a video of what he does when you're in the pen with him.