Awwww...what a sweet face. Makes ya want to hug him.
Okay, OP. This is what jumps out at me, and I consider myself no expert.
Very tiny bones, weak pasterns (I'm sure someone will have a better description) and a very odd shaped (roached) back.. If you block off the head and shoulders in the second photo it almost looks like a goat's body. Under no circumstances would I be putting much weight on him especially if you are tall and on the heavier side. I would think it would be hard to fit him for a saddle. And in all honesty I can't say he will ever be able to be ridden except for a very light rider.
I would have a vet go over him for an honest opinion as to his structure and ability as to usefulness as a saddle horse. Maybe he could be a cart horse? I don't drive but I'm sure someone who does could offer advice here.
I see him as a wonderful pasture companion at this point. And that serves a very useful purpose to many. A kind and easy going nature means a lot.
Please take my opinion for what it is worth. I hope he improves with age for you but at age 3, well......
Looking at that face still makes me want to hug him though.
If you ever find yourself in a fair fight, it's because your tactics suck. ~ Marine 1SGT J. Reifinger