Any birthday party on a holiday is hard for all who want to attend. But you have to decide if bringing it up will be worth the outcome. Not knowing the other side, its hard to say yes talk to them, or no don't. Usually parties are planned ahead, so that should give you enough time to plan your schedule accordingly. I could understand if the party is a last minute get together, but if it happens every year, then you should how to plan for it. Golf courses will always be there; people can have a picnic any time. If it really bothers you, next year say you can't attend because you have prior commitments, but say you'll make sure to stop by the weekend before or after or send a card/gift. To me, it isn't a big deal and blowing off a relative's birthday party to go golfing is being selfish.
This is what happens when you have democrats in office