Old vermiculite (Pre 1985 or so) was almost always associated with asbestos, almost all new stuff is asbestos free (according to the manufacturers!) But I wouldn't stand in a fan blast blowing the dust off new stuff either.
FWIW as of 2014 it was still perfectly legal to sell Asbestos worldwide....including the United States, that may have changed but I don't think it has....
As of that time the regulatory bodies had taken an attitude that there was no realistic replacement for it, a fund had been set up to deal with the health issues (and by deal with, I mean provide palliative care) and the rest of the policy was "caveat emptor".
I would not breathe it in. But this also applies to find sand, clay, perlite, asbestos, cement, chalk, paint dust, wood dust, …
(I am not a good example at this, as my lungs have seen all of the above, but they are not in the best condition either.)
Any time I have used Vermiculite, pearlite, peat moss from a bale or any other Dusty amendment I wear a Respirator (that has canisters on the sides), Dust masks are far to leaky around the edges and they don't filter out the particle sizes that are most harmful to your lungs.
Norton and some other companies make not so expensive models and some of these are available at most hardware stores or lumber companies, even Harbor Freight carries some that will be fine for our uses in gardening.
Eye protection is also advised when using dusty amendments since fine particles like to enter our bodies through them.
Vermiculite is a good amendment since it will hold water well over a longer period of time and it helps keep the soil open for air and water infiltration.
As others have mentioned, a lot of the paranoia comes from incidents in the past. It's not dangerous, but does give off fine particles.
Vermiculite mixed with Brown Rice Flour is the gold standard, because it has such a high success rate, for anyone new to mushroom cultivation. (to be clear, this isn't how mushrooms are grown commercial or even small-scale)
"Our ability to change the face of the earth increases at a faster rate than our ability to foresee the consequences of that change"
- L.Charles Birch
So I left, I came home, and I ate some pie. And then I read this tiny ad: