The area around my house in northern NM is filled with 6-12 inches of crusher fines. Underneath the crusher fines is compacted clay (which is why I put the crusher fines down, so in wet conditions I wouldn't be a muddy mess). Not much grows there but some native plants are starting to push through. The surrounding land is j/p forest with sage, along with some ponderosa pine and gamble oak along the mostly dry streams and a mostly unused irrigation ditch that goes through my land. I want to build rich soil there using permaculture practices. Should I remove the crusher fines or just leave them and build the soil on top? I do know that the fines are probably a great mineral source, but the crusher fines are the rock dust mixed with pebbles. Thanks very much.
First I would make sure that the area around the foundation drained very well so your home is not compromised by water infiltration. You can do this with a foot wide "buffer zone all the way around the foundation.
For places for garden beds, I'd remove the crusher so that it was no more than 4 inches thick in the areas where you are going to put compost/garden soil for plantings.
Build your soil on top of the left behind crusher and make sure you get a thriving microbiome living it that soil.
From there it is only a matter of planting the things you want to grow in those spaces.