Greetings Shane! Welcome to Earth Bag Central.
Not sure in your research if you've discovered Kelly Hart's site. Full of useful into and has a great FAQ section. This one on scoria as a fill material:
scoria
I'm not sure if the material you mentioned has the same properties of scoria: light in weight and full of small air pockets making it a great insulator.
I know scoria has often been used for the first foundation layers which is not affected by ground moisture, then continued with earth fill which contains up to 30% clay.
You'd probably want to look at a layer of insulation as well since the earth wall is great as a mass but not so much for keeping the cold out.
Some people have used insulation on the north side of the building and switch over to high mass on the south side.
Let us know what you find out and if you could give us a rough drawing of what your looking at as a building, we will have more to go on. Also a picture of the rock you have in mind as there are people here that may be able to identify what you have and its feasibility as a fill material.