Definitely not! Ponies are built for bearing a lot of weight. I'm 5'10" and 115 and I broke my 10hh Shetland. I've also been known to sit on my 8hh miniature once in a while
. As long as it doesn't feel like he's struggling to hold your weight or have obvious reluctance to trot/canter, you're fine.
BTW I love your avatar!