Since you only ride once a weak, it is hard fore your body to really build up the muscles you use in riding and not in walking or other daily activities.
It is important to have decently strong ankles for reasons other than riding. As you get older, if you don't have good strength in your ankle, foot and calf, you will lack stability and are more likely to fall down.
Work on building this stability. Stand on one foot, like a stork, for as long as you can. Really work on balancing using your core muscles and the many small muscles that surround your ankle. You can look down and you will see your ankle/lower calf making many small and constant adjustments to maintain your balance. Try to stand on one leg for a full minute at least without touching down, tho if you need to momentarily touch down, go right back to balancing.