The time to think about this was BEFORE you married him. People get married and never think about what's life going to be like down the road. Any habit they have before you marry them just gets 3 fold worse and at the end of the day life will be miserable down the road when you are old with no common interests
I'm sending you some prozac.
OP, do not be disheartened.
My father was exactly the same. My mum worked with racehorses at 16, then gave them up and joined the army. I was 14, my sister 13 and my mum 44 when we started riding again. We caught the bug and we spent our time down there.
Initially, my dad was supportive, then enjoyed his fishing time, then started wondering what all the fuss was.
He rode for eight years before retiring his horse last year due to navicular.
If he isn't interested, then don't push it. You'll just have sour feelings.
BUT if you're using the "us" funds to finance the future ponio, you REALLY need to do your research. How much the horse will cost initially, including all tack, monthly, vets bills, trainer bills, farrier bills, and unexpected vets bills.
If you can do that without struggling to feed yourselves because you have a nice horse, then go for it.
As a couple you don't have to do EVERYTHING together. You are individuals. I'm sure he will find his own hobby, or take one back up and you can share stories and talk.