At the latitude where I built an eath-sheltered home, the earth below the frost line was a pretty steady 52°. I insulated all 2-1/3 buried sides × the roof, which was covered with 5' of soil, using 2" Styrofoam board. I also insulated below the footers and the entire slab.
That may seem like overkill, but in 18 years there was never a moment's regret. It's not as if you can retrofit insulation below grade later.
I installed a ERV in the "attic" with ductwork along the north wall to the outside east. I had to wrap the ductwork along that run to prevent condensation, which would have led to mold, the bane of earth-sheltered homes. It was surprisingly quiet. There were no drafts anywhere, so it was comfortable even at much lower temperatures than one would think.
Best of luck on your build!
Lucy