Yes, saddle can be a factor, but I think personal anatomy and position may play a part too. I know riders who are very fit (in ways not involving horses), yet who are sore after riding. On the other hand, I am not really that fit, and I can do pretty long rides without really being sore afterwards, and I only ride maybe once or twice a week these days. So yes, your position may be having an impact. Are you riding with a coach who can correct it? A seat saver may indeed help. And trying your stirrups at different lengths.
One thing I cannot stand is a hard Western saddle. It kills my inner thighs (must be the twist). I've ridden in some really cheap, hard English saddles and was fine, but a bad Western saddle, I can't stand. So I do think the combination of saddle shape and body shape make a big difference.