Some thoughts from someone on a similar path. My wife and I don't have children yet, so that piece is different, and we live in suburbia, not the city. But - our plans involve moving back to where her family lives and building our own place in a rural(ish?) location.
So, a bit less to deal with in that we don't have three kids, but parallel plans.
My intentions are to build animal shelters and outbuildings before we start building our house. Experiment a bit, get some experience under our belts - and then get going on the human habitation. Along with just plain getting some practice in, we think this route will help us in dealing with the necessary governmental approvals. Alternative building is not particularly well supported under "code" standards and being able to offer some examples of working structures
should be helpful. At least we hope so. Most importantly, it gives us practice before we are trusting our lives, and our future children's lives, to our building skills.
I would have very real concerns about the child protective services reaction to children living in a shed. Our plans tend more toward a ger (aka yurt) as a temporary dwelling while we're getting the permanent home built.