MisssMarie - I'd not do that, for fear of getting prosecuted or charged with assault. A kick is more likely to be seen as a reflexive - not-premeditated - reaction.
And frankly, although I am female, and only 5' 6" I can be as intimidating as any man, it's all in the attitude. I tell people I may not have inherited my Dad's height, but I have a 6-foot-tall attitude regardless. (I did get at least 5 extra inches of height from Dad's side, not complaining!)
Nuala, keep your guard up, and try never to be alone with the guy, just in case.