Photos would help.
As for insanity, thats harder to
answer!
I would not demolish, I would rebuild.
Most places that want building permits allow renovation and repair without a permit.
If you demolish you may need a demo permit and then have to apply with drawings etc for the replacement.
I would pull one wall away at a time, rebuild properly and move ahead.
Cob, mudbrick etc are all good.
What is the original one built from?