That's some nice dirt! I'd give half my favorite tonsil for soil like that.
From my
experience, worms will find organic matter when you lay it down. If you really have a wasteland, it'll probably take longer. I think a lot of the worms in typical lawns have been poisoned or pushed out by lack of food and temperature extremes, but the little guys are resilient.
Something I found attracted them like nobody's business: rotten straw. Also, piles of rotten leaves, decayed wood piles, rotten cardboard boxes... look in those places and you may find some you can transplant to your beds. Old trees with deep shade beneath them and plenty of humus in the ground are nice as well. Shady, moist, cool corners are worm havens.
You could also buy bait worms and put them in, if you want to. They'll stick around if there's plenty to eat (rotting organic matter) and it's moist. It takes a while for them to build up a population, though. My worm population when I sheet mulched in TN didn't go crazy until about year 2. Then they were everywhere... and the work they did beneath the surface on formerly hard clay was impressive.
Good luck!