Yep, you look just fine on him. After riding something big, it truly is a big change to go to a small horse. I've been riding my 16.2 hand 1500 pound draft cross a lot lately and he's the only one I've been riding. I hopped on my Mustang the other day (he's 14.3 and 1100 pounds) and I felt like I was riding a miniature LOL.
It will just take a bit of getting used to. Since he's only 5, he could very well bulk up some more over the next 2-3 years. I had one horse that kept getting bigger and bigger every year until he was 9.