Don't get me started on cheese, we could be here a long time! ;-) Love the stuff. A friend of mine has Nubian goats - their milk is actually really nice to drink - not very goaty at all, unlike milk from Saanens and the usual European milk breeds. It doesn't spoil the tea, or the hot chocolate!
Goat breeds vary in the creaminess of their milk (just like dairy cow breeds, you either get quantity or high butterfat, not both). The alpine breeds are quantity milkers -- Saanens, Toggs, French Alpines, Oberhaslis. They are the Holsteins of the goat world. The African-derived breeds are butterfat milkers -- Nubians, Boers, Nigerians, Kinders (a dual-purpose cross between African Pygmy and Nubian with the highest butterfat of all). Those latter breeds' milk tastes a lot 'sweeter' because it is much creamier.
The other big factor in 'goaty' flavor in milk is proper milk handling. Raw milk must be kept extremely clean and chilled down as fast as possible (in an ice bath if you are a home milker). Otherwise it is going to start tasting goaty.
Oh, and keep your buck far downwind of your milking does!