I agree with both sides. I surely don't enjoy when someone who is a fanatic about stamp collecting wants to go on and on about the latest 1982 stamp they found.
But, I take offense when I feel criticized for my feelings towards riding and my horse.
My husband recently called my time at the barn an obsession. To which I promptly responded,
I have an obsessive compulsive behavior. You knew that when you married me. I can spend time at the barn or at the casino. Which do you prefer?
Yep. Just what I expected.
I have also found that most people who use the word "obsessed" do so not because a person is boring them, but because they feel the person should have more than one interest. That if you only spend your time with one interest or hobby it is an obsession.
I have also discovered that the people who use the word obsession in regards to other people's hobbies tend to be "watchers" rather than "doers."
For example, I have never been told I am obsessed by another person who has a passion that requires either training or skill. People with a hobby that requires you to master something appreciate the hard work it takes to nurture a passion.