DE has edges like sharp knives, and worms stay away from that. There is that science tidbit about things break down in the same shape no matter how small the particles. So if something appears in angular pieces with sharp corners when smashed, no matter how much you smash it, those pieces will always be that shape.
If the worms are surviving, then they are finding soil to pass through themselves that doesn't contain it or contains very little, which is the only way they eat. Or they are finding
enough of what they need to reproduce, which is quite quickly, something like 4 to 6 weeks? and it looks like they are surviving, but it's new worms.
If DE is in the soil, or egg shells, or sharp beach sand, worms will avoid them. You can't take them back out of the soil, so they are a long-term thing. There are other soft-bodied soil friends we need who also stay away. They affect the balance of things. It isn't just about worms.
An important distinction: Permaculture is not the same kind of gardening as organic gardening.
Mediterranean climate hugel trenches, fabuluous clay soil high in nutrients, self-watering containers with hugel layers, keyhole composting with low hugel raised beds, thick Back to Eden Wood chips mulch (distinguished from Bark chips), using as many native plants as possible....all drought tolerant.