This is the great "Trail Riding Dilemma" and why I mostly ride by myself. I have friends who only want to walk on trail rides and I like to school though all the gaits and various things I want my horse to know.
When riding in a group, I look at it as social time and we only go as fast as the slowest/greenest horse & rider are comfortable with. I can do some things to ease the boredom but mostly I'm just there for the company and go along and get along.
When I want to work on stuff, I don't even really want anyone to talk to me so riding by myself works best. My friends all know I'm like this no one gets their feelings hurt. I also try to pick times when they aren't available to ride anyhow to go do my schooling.
The main thing to remember is, just because someone has a horse, it's not your responsibility to keep them entertained. He's a big boy and can find others to ride with if he really wants to. It's your off time and your time together, you & your hubby, to be with your horses and have some down time as a couple too. You don't OWE him that time.