I don't know anything about radiant barrier insulation so my comments may not be totally useful. As I'm doing math on the heat loss for my future greenhouse, it's amazing how horrible the heat loss is through the glazing vs the insulated walls. Taking the walls up from R20 to R40 is a drop in the
bucket when you see how much heat is pouring out through the glazing. In my case with R20 walls and R1.4 glazing I'm going to lose 3,000 BTU/hr through the walls and 24,000 BTU/hr through the glazing. Doubling the wall insulation to R40 would save me 1,500 BTU/hr. Doubling the glazing R value would save 12,000 BTU/hr in comparison.
I'd recommend reading Bioshelter Market Garden by Darrell Frey. He puts a lot of awesome calculations in the book that you can use to determine the heat properties of your design.
My struggle with insulation is the moisture in the air. I'm not sure if there are any good options other than styrofoam or spray foam. Fiberglass would likely get wet and stop working. I'm very open to suggestions in this area.
I have a couple other thoughts on your design (as I understand it). First is that the "below grade" section of soil will be pretty cold so I'd insulate that from the interior. Frost goes a few feet deep and having frozen soil against your foundation or wall will cool the interior. Secondly, if you're intending for the thermal mass to pick up
solar gain, it has to see the sun. So tall plants may shade it. It would still warm up from the warm air of the greenhouse but that gain may be less than you are counting on. I'm tempted to put my thermal mass in the shade in my greenhouse but plumb the tanks/barrels all together and circulate the water to a big radiator in the ceiling where it's getting blasted by the sun and is at the hottest part of the greenhouse.
Sorry it's taken so long for you to get some input on your design. I read your post a month ago but was hoping others would chime in that I could learn from
