Hey there,
It sounds like your soil is heavily compacted as well as being clay.
One of the key problems that leads to shallow roots is compaction that stops water, air and nutrients from getting into the soil. If the soil is pretty squashed they will stay around the surface so you need to aerate the lawn before you do anything else (including worms - who won't won't to hang around in really tough soil).
As it sounds pretty severe, you may want to get yourself a
drum aerator that pulls small cores up out of the soil and both breaks it up and also encourages better root growth. You will end up with thin plugs of earth on the surface but these will break up quite quickly and help to decompose any organic matter on the lawns surface over time.
You may need to do this once or twice over a few months depending on the soil and then overseed and water and feed to get the lawn recovering. Once on the road, keep it aerated by using something like
lawn aerator shoes so you get a constant but lower level of care every time you mow or just fancy a walk!
Good luck! Jack.
