You definitely want to be straight up, because saying to keep it between you he could interpret as consent and to just keep it quiet... Sadly, sometimes even a straight out "NO!" is not understood by men, but you have to be frank. Any leeway and they can interpret it their way... I once told a guy I hated him and wanted nothing to do with him. His answer? "You're exaggerating!"
You probably also want to make your husband aware so it doesn't come out later and he'll feel betrayed that you didn't say anything. Depends on your relationship too though. That is what I would do with my husband, but we are very honest with each other and I have a low tolerance for unwanted male attention
Hope you can get this resolved. It definitely sucks to be in that position.