That would be extremely difficult to do. Even if somebody were able to get the ip address that an email originated from, in order to find out the physical address of the person using that ip address they would have to contact the Internet service provider that assigned the ip address, find out which customer the ip address was assigned to, and then track down the physical address of that customer. Most Internet service providers take their customers' privacy very seriously and would be reluctant to release any information of that kind. In fact, there may be laws against it and they may only release it by court order. (I say may because I don't know, but that would be my guess.)
While you can do a geographical lookup of an ip address, it only tells you the approximate location of the Internet service provider it was assigned to, not the customer whom the Internet service provider assigned to it, though with local Internet service providers these are often one and the same. Even then, though, you're only getting an approximation of city, state, and zip code.
You can prevent even this by doing your web surfing through a proxy server that replaces your ip address with it's own... if you're concerned about this sort of thing.