If you don't want to ride them that is YOUR choice, no one else's opinion should matter.
My step mother had polio, as a kid, it in effect left her unable to ride horses, but she loved them. Ultimately her love for horses, got my dad into it, so from the time I was 4 we have always had a hand full of them around. She would sit on the porch and watch them on pasture, she would go out and give them treats, and lead them, and groom them. She just did not have the leg strength to ride.
Now, my father and I rode them, but if we did not, she still would have had them.
I take the horses Im not riding in the summer and pasture them at my parents so that she can watch them eat. I think there are several people in your boat.
I personally would probably not own horses, that I did not ride, the day to day care is basically just a means to an end for me. I don't particularly enjoy busting ice and hauling water all winter when it is cold, or sweating my *** off in the summer moving hay, but i do enjoy playing with horses, so I just accept it as part of the package.