I am an adult, I own my horse and I don't even ride every week. You are actually being incredibly responsible and cutting back when you need to.
Yes you would get something from just riding once a week, heck you'd get something from just riding once a month. You don't have to ride to be around horses if you can get yourself to the barn. Lots of stuff we learn is learned in the barn rather than the arena.