Hi Colin,
the way I understand it you want high thermal mass and high conductivity around the tubes. The air flows through the tubes, exchanges heat with the tube and surrounding "dirt" and gets warmer (at night, in winter) or cooler (summer, midday). So you need material that absorbs and hold on to heat.
The condensation that you're talking about will happen when you cool down hot (moist) air and will happen inside the tubes, not in contact with the backfill. So I don't see why it shouldn't work with heavy clay. It might actually be your best material.
But maybe I'm misunderstanding the whole setup. GAHT didn't get me many results googleing.