I did pony club when I was in high school, and I never really "fit in". I enjoyed it for what it was and never tried to be best buds with everyone. I started riding when I was 11, and took lessons until we bought my horse when I was 15. I joined pony club at around that time. There were basically two groups of people in the club- those between my age and 4 years older than me, and younger (lower and middle school) girls. I didn't jump, hadn't competed heavily in my younger days, and wasn't one of the "grand" riders, so I didn't really fit in with the older girls. Had I turned the clock back four years I probably would have gotten on great with the younger girls as a new, young rider, but I didn't exactly enjoy spending all of my time with 8-12 year olds. I went to the meetings, was stable manager at a few rallies, and did some great clinics with the group. It was definitely worth my time and taught me a lot, but I didn't make any lifetime friends doing it.
There was also another local pony club group, but it was very small and basically only had a few main members including one of my best friends. When I decided to join the barn owner, my trainer, the lady at my barn that ran my club, and everyone else pushed me into joining the "bigger, better" club. I probably got more out of it than I would have from the smaller club, but probably would have enjoyed the smaller one better. Since then, I think the bigger club moved to a town an hour or so away when the lady that ran it got her own property. I don't know for sure, but I think the smaller one sort of fizzled out. Sad times for pony club around here
Moral of the story: Give it a try. Even if you don't fit in, you can still get some great instruction from pony club. Try it for a year, and if it isn't what you wanted then you don't need to come back next year.