I didn't read all the posts from other people, so may be repeating thoughts, lol. You are the one that got yourself dumped, not your horse. Do not EVER get on a green horse when they are already using signs to let you know they aren't wanting you on. Your horse was telling you strongly that he wasn't ready for you to get on. If he ever won't stand still, you need to move his feet until he decides standing still might be the best option for him. Also, with a super green colt like he is, do not change up what you've been doing with him, ie. Getting on without lunging.
When I broke horses for other people, I'd tack them up and let them hang tight on a pole or the walker, then work them until they were attentive(The most important part), then would go about making sure they weren't going to dump me or act silly, like flap the sturrips and pull on em, pat all over, make noise, jump up and down beside them, flex them, etc. Getting on a green colt cold turkey is just asking for trouble as you haven't warmed up their minds and gotten them ready to work with you.
Everyone is right that you let him win 100% that night. Not working the sweat out of him and walking/lunging him UNTACKED did absolutely nothing. If the situation ever happens again, get those feet MOVING like you mean business, Immediately. Do not stop until he shows that he is ready to work with you, then do a couple groundwork exercises/desensitizing, do all the prep work to get on, THEN and only then, get on. I also never get on a colt the first few rides unless he is flexed towards me. As they progress, ill flex them less and less til they aren't flexed at all.
Remember that as nice and cutesy as you think this colt is, he just proved he is more than capable of hurting you and will, if you don't immediately correct him now, before he gets much worse. 5 rides mean nothing on a colt if they weren't productive rides where he wasn't solely attentive, being respectful, and listening to you. Green riders on green horses tend to lead to trouble more often than not. If this is your first time breaking a colt, get a trainer to help you break him. You'll have a nice riding horse in no time =)