I have similar issues when I dug a hole, put a tree in with some nice soil and amendmends. The problem may be that such trees stay rootbound as they don't want or can't get into the hard clay and stay in the nice soil. In addition, rain does not drain as well in clay, so it is stuck in the nice soil perhaps rotting roots so the tree may be working hard putting out new roots to replace the ones rotting or diseased, I planning on addressing this this year thanks to the idea from Marco Banks:
"One example: I had a failure-to-thrive cherimoya tree in my orchard. Whatever I tried, it just didn't seem to grow very well. I mulched around it regularly, but it just didn't want to grow. It had been in the ground for 8 years, but very poor growth and hardly any fruit. Four years ago I planted about 20 comfrey plants around it, starting at 2 feet from the trunk and then out past the drip line. Very quickly, the comfrey filled-in and created a green moat around the little struggling tree. Twice a year, I chop and drop all that biomass around the tree. The first year, not much difference. The second year, the comfrey had fully grown in around the tree and it started to look much better --- healthier leaves, more blossoms, and some significant growth with the branches jumping up about a foot. Last summer, that tree took off and grew a couple of feet. Without going to heroic lengths to pollinate (which you'd understand if you grew cherimoya), we had a dozen nice fruit on it. This summer, it's grown about 3 feet thus far and probably will grow at least another foot or two before it drops its leaves in Feb. Nothing changed in terms of my care for that tree other than the addition of the comfrey. But the effect upon the soil around the tree is stunning --- black, crumbly, full of fat worms --- and there are about 30 ripening cherimoya hanging on the branches. "
Hope it helps you as well