https://en.wikipedia.org/wiki/Artisanal_mining
I have no land and haven't done anything, but I've toyed with similar ideas. In Arizona, where I live, there are lots of private mines that open up to rockhounds to go out and dig pits for fire agates, for example. I know there are many similar options for all kinds of materials. Slate for example. I've hiked many places that had vast amounts of usable slate just lying there. I'm nore interested in natural pozzolano. How do get it out in the quantity you need, etc, is the obvious problem.
My plan it to build with adobe, given the climate. I'm expecting to use the adobe pit on site as an irrigation cistern, or possibly a partial basement or sunken courtyard.