Give it a couple more rides, you'll get used to the feel of him. =) At one time I owned two horses: a 14.2hh Arabian gelding and a 16.1hh OTTB gelding. Now THAT was weird! LOL I'd go from riding my Arab to my TB, or vice versa, and be like WHOA, where did my horse go, or wow, the ground is far away. Hahaha. Not to mention the different in gaits from one to the other. Riding different horses is great though. I find that I get "stuck" with one particular horse very easily. If I ride it for a long time and go to another one I'm always a little uncomfortable on the new horse initially because I'm SO used to my old one.
Just give it some time. Take some deep breaths, relax and just walk around a little to start. Get used to the height difference, the feel of a longer stride, the way your leg feels on him as opposed to how it felt on a pony's barrel, etc. When you ask for a trot go nice and slow, stay collected and relax. Once you are used to his rythm and comfortable with it then canter. =)