You are for sure too tall to look right on your pony.
I think your saddle problem is a combination of things. Yes, it is too small for you. But the fact that it obviously does not fit your pony right so you end up way shoved to the back of it (actually hanging off the back of it) really does not help at all. Is your saddle way too narrow for your horse so it perches up on top of his withers?