Recognising the issue of tons of work is the best start you can make.
I have been involved with earth building a long time.
There are different ways and each has its merits.
With rammed earth slow construction is a problem.
You could look at earth bags, compressed bricks have one advantage, if you build the roof structure first and plan on in-fill walls, you have a shady and dry spot to work under.
As you make the bricks, they can be stacked out of the weather and the roof will catch
water for the production process.
With the roof, no time is spent weather protecting the walls.
Think about that.