In addition to the ones we had around the house removed we had one that was very close to our barn. The roots had grown into the stalls. I wasn't worried about the tree coming down on the barn but it worried me that if lightning hit the tree the lightning would run right into the stalls. Down the tree down the roots in the barn.
We still have some really large ones around the barn and the house but none that close. If one of these fell on the house I doubt there would be anything left including us! Sure does make a difference in the summer time. The shade is lovely.
Just DO it and be happy that you can!