Sounds to me like you haven't really developed a bond with this horse. Horses that like their humans will leave the herd and come to them voluntarily. If every time you catch him it is to ride him, you haven't spent a whole lot of quality time with him.
It doesn't sound like he's really dangerous. You're just on the same learning curve as all beginners are. He was probably a bit young as a first horse, but you have him now so I wouldn't necessarily give up. He's probably filling in alot for you (i.e. taking care of you) so maybe he can be a good teacher for you.
Why not spend more time with him and see if you can develop a better rapport? Learn how HE communicates so you will notice things like his ears and which way his attention is focused (should be on you). I agree with finding a good instructor. Someone who can work with BOTH of you together and teach you what you need to learn so you can catch up with your horse. This should start on the ground, not under saddle.
Good luck and I hope things work out.