Your mix may be OK, but it needs a binding (or re-enforcing) fibre. In traditional cob, straw is usually used as the 're-bar' that knits the monolith of sand and clay together. Try some test bricks with chopped straw in the mix, also try a few differing ratios of sand and clay, see if you get any better results.