Joel Hollingsworth wrote:
 I don't usually, especially if it's moist.  The hazardous stuff isn't available anymore, and from one mine to another it really is like night and day as far as the hazards.
 
 If you happen upon some many-decades-old stuff and can't look up where it came from, treat it like asbestos.  Wear a mask and coveralls and gloves, use it for masonry not gardening, spray water around...or pay to have it disposed of properly.
 
		
  The danger with vermiculite, perlite, pumice, etc. is not just rare minerals.  (I haven't heard about the specific mine problems, so I'm just guessing - was a toxic mineral part of the problematic deposits?) 
 The other problem is always present with masonry insulation: plain ol' ordinary mineral silica.  The same foamy texture that makes these rocks so insulative also tends to create sharp little shards of dust, which will damage lungs quickly.  It's like glass powder. 
 Some people complain that someone should have TOLD them that all rock powder can damage lungs, or that rock with sharp little faces (some crystals, or foaming processes, can produce hair-thin needles of silica) is even more damaging. 
 So now I've told you, and you know.  It's just a stronger degree of the same hazard.  And it's what makes the Public Panic Asbestos Removal Program so ironic, because some untrained people are deliberately ripping out asbestos in fear, when in fact it's much safer in place and they're creating dust by ripping it out.  Trained teams with dust-handling equipment can rip it out safely if you need it gone from a project; otherwise, just DON'T DISTURB IT and it's inert. 
 I try to wear a mask when handling dry masonry materials, whether it's clay, lime, or especially the insulative materials like puffed rock or fiberglass/mineral wool.  Anything with a small enough particle size to be poofing dust when I handle it, is not something I want in my lungs.  Potters, masons, miners, even jewelers can get silicosis from job exposure. 
 When a mask is not available, or in case there are unmasked people nearby, dampening the material like you said is a great option.  We sometimes spray water directly into the bag before dumping out the perlite or vermiculite.  Being outdoors where the dust has a chance to settle onto something other than lungs is a good thing, too. 
 -Erica Wisner 
http://www.ErnieAndErica.info