This forum isn't just for riders, though! And it doesn't make you any less of a horse lover if you don't ride. In my opinion, riding has nothing to do with being a good horse person.
You know Bob Langrish, the famous equine photographer? As much as he loves horses, he will not ride them. Go figure.
I think I might also be drifting away from riding. I have no one to ride with, no trails within riding distance, and no horse trailer. So it's kind of pointless. Also, my horse went lame the first week of July, and it scared me so bad that I haven't ridden him since. I know it's ridiculous, but I'm paranoid.
You're not alone. And don't lose interest in horses just because you don't ride. That would just be silly.