Smoothness of gait has nothing to do with size, it's determined by build. I've been on giant horses that felt like jackhammers and some that were unbelievably smooth. You do have to post much faster though on the little ones. :)
I know exactly how you feel though. 95% of what I ride are ponies, but this past year I had 2 horses I needed to show so I had to get used to riding the big ones again. I think the biggest differences are: size of barrel (you have to almost reestablish your leg when you go from the big ones to the little ones, or vice versa), turning radius (just like driving a really big car or a compact. even if the big ones turn very easily it's not like turning on a pony! good thing is, it's harder for the big horses to suddenly turn out from underneath you like some of the ponies can), and length of stride. This is the biggest change when it comes to jumping. It takes a bit to adjust to adjust to finding a distance again. But it WILL happen! You just need time and practice. If you ride nothing but big horses for a bit, you'll find that sitting on a pony feels kind of strange. If you're wanting to show in the 3" yea, it's about time to make the switch. I know the larges jump 3" at rated shows but I think there comes a point in most people's showing career when they need to get a horse if you want to progress. Plus, you can't show in the pony classes past 18 anyways.