You MAY be leaning to the right as you ride and don't realize it. Are you right handed? I know I used to lean heavily to my dominant side starting out, and I watched my son and daughter, early on, as a ride went on, start to lean heavily to the right - they're both right handed and so am I. Son still does it - I have to tell him to square up or he's going to fall off every now and then, which is weird because he still can ride like he grew there. Just makes me so aggravated he doesn't take more interest in it.
Anyway, I kinda feel as if as you get tired, your weight will start to 'sag' if you will to your stronger side, then you're pushing into the stirrup to support your weight, then your ankles, knees, then up through the hip start to hurt.
I can also verify the sore knees thing in that cowboy saying. My knees used to SCREAM after only riding a couple of miles. It felt so horrible, only riding that far, and aching that bad in my knees and I'd feel my toes curl from cramps, or a charley horse would start in my hamstring or butt cheek... hard to control a horse when you're hurting like that.
Then I started letting out my stirrups, a hole here, a hole there... it got better. I have them out real far now, but I think one more hole would be too much. Rode a saddle not mine late last summer, a high back ranch saddle I'd brought camping as a spare, bought it to flip it, thought I'd ride it to see how well made it was.
The stirrups LOOKED like they were way out, but they weren't. Gina was kind enough to let me let them out while I stayed in the saddle, and they felt okay... two hours later, my knees were on fire, felt like I had ground glass in them. I'm thinking I needed two more holes let out on the stirrups, but by then I'd decided I hated that saddle and was glad to see it go when it sold.
I think you'll find as time goes by, and your body gets conditioned to the changes in stirrup position and seat, you'll find you're not fighting the stirrups as much. I don't know what kind of saddle you're riding, but a lot of folks try trailriding in roping saddles. I was one of them, didn't know any better. THOSE have very wide, very thick stirrup leathers, thick fenders. The entire saddle is over-engineered from a trail riding POV because it has to withstand the torque of an upset steer, calf, cow, heifer, or bull being tied off to the horn and the rope snapping tight. The roping saddles are what killed my knees in the beginning. Then our dozer operator brought over his daughter's SRS barrel saddle. It was ugly - the seat was kelly green suede, but the difference in the seat position, the feel of the horse under it, how the stirrups swung much more freely, made a convert out of me.
I ride trails in a barrel saddle now. I have a used Saddlesmith made Connie Combs I paid 200.00 for, intended to flip it for a profit... and it's never left my house. It's only saddle I've had so far I refuse to sell for any amount of money.
My husband rides in an SRS ranch cutter - for the same reason. Easier on the knees, stirrups swing freely, he likes the seat position, feels more horse through the saddle and it's comfy for long rides.
I hope this helps you - I've had to learn mostly on my own, though I can thank a lot of young people for the early days and helping me get started... but some of them were very wrong about what they taught me, and it's because they were taught wrong. Trail riding has been one of the most amazing things I could do with my horses, and I am a huge advocate of camping with horses and trail riding - even if you run barrels, rope, anything organized in an arena - it's good for your horses to give them that change of scenery. It makes them think differently, it gives them a workout they would never get in a level arena. It can make you and your horse learn to rely on one another and trust one another on a much deeper level. Wildlife doesn't spook from horses like they do from people on foot or motorized vehicles, btw. I've seen deer, pretty garter snakes laying in the sun on a log, birds of all sorts, foxes, turkey, geese, ducks, raccons, hogs (Nasty sons of bucks... Not a fan of wild hogs), all sorts of critters that I'd never normal see if I'd been on foot.
"We are here to laugh at the odds and live our lives so well that death will tremble to take us."