Hi Tim. Welcome to the Wild World of
Rocket Mass Heaters!
Your question has been a common one over the years and has had many takes on its performance.
Generally speaking rocks,
pea gravel and sand, no matter how densely packed are inherently surrounded by tiny air pockets which act as insulation.
What your looking for is a solid monolithic mass free from voids or air spaces as much as possible.
Its not to say that you can't use a loose fill (for a quick and/or portable installation) that allows air to percolate up through the pebbles, but again, its performance at retaining heat is not nearly as good as a solid mass.
So you could still use the rocks, stone, gravel you have access to, but it would be better to fill all the voids in with a clay mortar. Go heavy on the rocks and just enough clay/cob to join it all together.