As for the fear of having it taken away, like others said, training is the best answer to that. Because of my previous experience, I've had classes in weapons retention that taught me how to keep someone from taking my gun away from me. IMHO, that is an absolute necessity because you can't always get off a shot before they get to you. The average assailant can cover 28 feet before the average gun owner can draw, get a line of sight, and fire a single shot. The problem with that is, most folks don't even realize they're in danger until they've already been grabbed.
For that reason, some form of self defense classes are an absolute must. Not only will that give you an idea of what to do when/if you are assaulted, but it will teach you confidence and spacial awareness that will make it less likely for a perpetrator to surprise you.
Talk to your local law enforcement. They might know of somewhere that teaches self defense and weapons retention classes. If all else fails, get to know a couple of cops and ask them to teach you.