You're probably back at it by now, or soon...
Have you considered putting the roof up now / soon, at least a temporary roof, so you can
shelter the building site and keep building in various weather?
A roof and some tarp or plastic walls, like greenhouses have, can extend frost protection several weeks.
It looks like your walls are thick
enough that they could compensate for a little frost damage. I've seen sample projects loosened up over time by weathering and freeze/thaw cycles, particularly water crystals in my clay storage barrels are pretty interesting, shredding the clay slip into angular curds.
Have you discovered any differences between the layers that froze while wet, and those that didn't?
-Erica W