Hey TJ, an interesting dilemma. One approach might be to bite the bullet and build something really small and basic and “to code” to get the occupancy permit, and then get to work on the structure you really want.
This has a couple of advantages. First, it can keep child welfare folk off your back. Second, it improves your prospects for resale should you need an exit strategy. An alternative, non-permitted structure will really be viewed as raw land for most people. Always good to have an exit strategy! Bad neighbors, illness, disability, family issues - all might require a move someday, and no matter how much heart, soul and money you pour into it, if a purchaser can’t get a loan, no sale.
It might be worth undertaking an education campaign of your local building inspector. Might take time, and might require some compromises, but at the end of the day most code requirements originate from safety concerns. Even if you are willing to live with it, they need to “protect” those that come after you. Not sure what county you are in, but some are more “active” than others in enforcement. I do know someone who built a straw bale house, and one who built without a permit. In your shoes, with kids, I would be pretty reluctant to risk a no-permit build too.