Cob Oven - Adding Rock to Cob Mix

I am looking to build a cob oven. I have a ton of small landscape rock (marble size) laying around that I am looking to get rid of in a productive way. I figured that I would add some of the rock to the insulating layer of the cob oven because the rock will help increase the thermal mass. I see it as an advantage to having it in the mix. Any reasons that I shouldn't do this? Are there disadvantages that I am missing? I realize this isn't "cob" in the true sense, so I don't want to debate over that. Thanks.

I think the rock will just take away from the thermal mass. If you want better thermal mass just use mud.
For the insulating layer you should make a clay slip and mix it with fine saw dust or perlite. Putting rocks in there just doesn't seem like a good idea to me...
But I'm curious what turns out.

Thanks. I guess there is the trade off between insulating and thermal mass. Probably not a good idea for the insulating layer.