I know what you mean sister!
My horses went on vacation for almost three years with daughter #1. And so did I (from horses). My horses live with me and I still didn't get to spend time with them very often.
Kid Number 2 just turned three, and yes again, they've been on vacation for the past three years. She is old enough to ride with me on the saddle from time to time but that doesn't count as real riding time for me.
With kid number one, I fretted and complained and basically made everyone's life miserable in my family because I kept demanding they take over so I could ride. When baby #2 came along, I just accepted that this is what it would be like - no problems and now my time is coming back.
Another thing I've observed -so many of my friends who've had kids that are now teenagers never got back in the saddle because their kids got into horses and they are now grounded being the 'horse mom'. They love it of course, but I told myself a long time ago that I didn't ever want to give it up...I was going to make sure I kept riding, even with the little 'time outs' that come along.
It's okay to take some years off to be a mommy. You'll still get to get on here and there, but there will be little time for a real hobby with it and that's ok! They will be there waiting for you when you return. I promise!