Goat can usually be found for a good price at one of my local mexican stores. The only thing that I can complain about is the variation in flavors you get sometimes. I think it's mostly because of the individual animals they use. Older, uncut boys tend to be kinda icky tasting...
I had lamb chops before when I was little along with rabbit. It was my Dad's friend who was renting out his old dairy barn to a lady that raised meat rabbits, goats, and sheep. We got alot of free meat if she didn't get it sold like she wanted.
Venison and summer sausage made from deer is my all time fave though.
Here's a website that has some good info on different types of meat.