He's a handsome fella alright! I think it's a great idea to keep him where he is so you can keep working with the barn owner to finish him a bit. It sounds like he could still use a little work, but that he has a good mind. A lot like my Rusty! Who is coming along very well. 5 hours in the saddle sounds divine! The most I've done was 7, and surprisingly, I tolerated it quite well. These days, I barely have time to ride for an hour here and there :(
Have fun with him, ride him as much as you can, and spend a lot of time just hanging out with him too! Congratulations!