My #1 suggestion would be to ride without stirrups. When I ride without stirrups, I first warm up, then take them off of my saddle. That way, they are not banging around on the horse's sides and they are not so easy to pick up when it gets difficult.
If you get tired, take a break and walk for a few minutes, but try to push yourself a little bit. When your legs start to ache, try going just one more time around the arena, then take a break. Over time, you will build up the strength in your legs, and, in my opinion, no-stirrups work makes you more aware of your seat as well.
I would always add that if you are in reasonably good shape and your legs hurt very soon after dropping your stirrups, you might be gripping too much with your legs. Allow the horse's motion and thrust to push you out of the saddle.
Also, besides riding, I also run, which is great for leg strength and cardio.
Hope that helps! :)