The problem lies in what happens when you sell your house and it falls on the next person. So while I don't like excessive regulation, it's a pain to remodel my house and find all the oddities (some fine, some very unsafe) that the previous owner did because he never went through the proper channels on permitting. But I think if everyone simply could say "I modified/built this house outside of spec" and you got a seller to sign and agree, then there should be no problem.
Outside of building codes, I think a lot of zoning is silly. I can have miniature livestock but not full sized. I pointed out to the city that it means I can have a 600 lbs mini cow but not a 150 lb full sized goat. They didn't see that as an issue or why I found it illogical!
