I've lived in passive solar earth bermed houses in the high desert.
What climate are you in? And you implied you're in the northern hemisphere, right?
1) In a humid climate, in summer the thermal mass can attract condensation. It wasn't a problem in our dry climate but I've seen it in another place I've visited many times: humidity to a problematic level.
2) The earth down deep holds at about the annual average temperature. Where we are, that is something like 10C, which is lower than comfortable room temperature. So there's a conundrum with the earth berming. It's much warmer than the outdoors air temperature in winter, which is good, but it's lower than the desired room temperature. So the best is to have the rooms you want warm on the south side, and use the earth bermed northern side of the structure as a thermal buffer zone, things like corridors, storerooms (great for a pantry I can attest!), stairs, dry composting toilet room.
3) Solar gain really truly happens best on the south-facing side. East, west and north facing sides get excess unwanted sun in summer and insufficient sun in winter. Overheating in summer, especially night overheating due to west-facing windows, is seriously uncomfortable. Your situation, with the solar exposure windows above the living spaces won't work well for passive heating, but could work for an active system using a medium like air or water to move the heat to the living space. It's not dictated by "solar police" but by geometry and how the earth turns.
4) If you're in a climate with plenty of wood and you like heating with wood, then the strawbale and rammed earth construction could hold enough heat to reduce your heating season and might be part of a solution.
Best of luck! Curious to hear what climate and latitude you're in.