I grew up with milk goats, when I was younger I even had my own registered herd name and a herd of registered Dairy goats. Not your average bush goats, my goats produced around a gallon a day each. I've raised Nubian, Lamancha, Alpine, and Oberhasli dairy goats, and some Kiko, and Boer meat goats as well. There are a few things that are misconceptions in this thread.
Goats are no more prone to problems with wet environments than sheep, both will get hoof rot from standing in mud and manure. As for altitude, Its never been an issue for me; I've had goats near sea level in Florida, in the high Texas desert, in the humid hills of Alabama and Georgia, and in the mountains of east Tennessee. I've pastured goats in forest, swamps, desert, and just your regular old fields. The did worst in grassy fields, goats need browse not grass.
Any strong flavored herb will come out in any animals milk. Even human. Farmers have for years eradicated green onions and garlic from their fields for this reason. Likewise if you leave the buck in the pen with the milking does you will have a bucky flavor to the milk. He only needs to be in there when breeding. In nature the bucks hang out with each other in the off season.
You don't necessarily need to feed grain to get good production, you just need high quality browse. Or good hay, Alfalfa is extremely high protein and will keep production up as will any legume that is high in protein.
I've had dairy goats individually and in herds in the teens and twenties.
In the end the choice is yours, if you have any questions I can answer let me know.