You're just fine. My Morgan was just over 14 hands and carried me fine all day long - mile after mile. I was 5'9 and just over 120 pounds at the time. It's nice to have a horse that you can just hop up on bareback.
Yep, you look just fine on him. After riding something big, it truly is a big change to go to a small horse. I've been riding my 16.2 hand 1500 pound draft cross a lot lately and he's the only one I've been riding. I hopped on my Mustang the other day (he's 14.3 and 1100 pounds) and I felt like I was riding a miniature LOL.
It will just take a bit of getting used to. Since he's only 5, he could very well bulk up some more over the next 2-3 years. I had one horse that kept getting bigger and bigger every year until he was 9.